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Lattice point problems and distribution 
of values of quadratic forms 

By V. Bentkus and F. G5tze* 
Abstract 

For (i-dimensional irrational ellipsoids E with d > 9 we show that the 
number of lattice points in rE is approximated by the volume of rE, as r 
tends to infinity, up to an error of order o(r'^~^). The estimate refines an 
earlier authors' bound of order 0{r'^~'^) which holds for arbitrary ellipsoids, 
and is optimal for rational ellipsoids. As an application we prove a conjecture 
of Davenport and Lewis that the gaps between successive values, say s < n(s), 
s,n{s) G Qi'^'^], of a positive definite irrational quadratic form Q[x], x G M*^, 
are shrinking, i.e., that n(s) — s ^ as s ^ oo, for d > 9. For comparison note 
that supg(n(s) — s) < oo and inf^ (n(s) — s) > 0, for rational Q[x\ and d > 5. As 
a corollary we derive Oppenheim's conjecture for indefinite irrational quadratic 
forms, i.e., the set Q\L'^] is dense in M, for d > 9, which was proved for d > 3 by 
G. Margulis [Marl] in 1986 using other methods. Finally, we provide explicit 
bounds for errors in terms of certain characteristics of trigonometric sums. 



1. Introduction and results 

Let M*^, 1 < (i < oo, denote a real d-dimensional Euclidean space with 
scalar product ( • , • ) and norm 

|x|^ = (x, j;) = + • • • + x^, for x = (xi, . . . , x^) G M'^. 

We shall use as well the norms |x|i = '}2'j=i \xj\ and |x|oo = max{ \xj\ : 1 < 
j < d}. Let Tj'^ be the standard lattice of points with integer coordinates in M'^. 
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For a (measurable) set B C M"^, let vol B denote the Lebesgue measure 
of B, and let volz B denote the lattice volume of B, that is the number of 
points in B nZ'^. 

Consider a quadratic form, 

Q[x]'^= {Qx,x), for xeW^, 

where Q : ^ denotes a symmetric linear operator with nonzero eigen- 
values, say qi,...,qd- Write 

(1.1) Qo = niin \qj\, q= max 

We assume that the form in nondegenerate, that is, that qo > 0. Thus, without 
loss of generality we can and shall assume throughout that qo = 1, and hence 

q>l. 

Define the sets 

Es = {xeR'^ : Q[x] <s}, for s G M. 

If the operator Q is positive definite (henceforth called briefly positive), that 
is, > 0, for X / 0, then Eg is an ellipsoid. 

Recall that a quadratic form Q[x] with a nonzero matrix Q = (qij), 1 < 
j < d, is rational if there exists an M G M, M 7^ 0, such that the matrix 
MQ has integer entries only; otherwise it is called irrational. We identify the 
matrix of Q[x] with the operator Q. 

Our main result. Theorems 2.1 and 2.2, yields the following Theorems 1.1, 
1.3 and 1.5 and Corollaries 1.2, 1.4, 1.6 and 1.7, proofs of which we provide in 
Section 2. 

Theorem 1.1. Assume that Q is positive and d>9. Then 

(1.2) supA(s,Q,a)= sup ' = o{s ), 

aeK'' a&d vol 

as s ^ CO if and only if Q is irrational. 

The estimate of Theorem 1.1 refines an explicit bound of order 0(5"^) ob- 
tained by the authors (henceforth called [BGl]) for arbitrary ellipsoids. That 
result has been proved using probabilistic notions and a version of the basic in- 
equality (see (3.12) below) for trigonometric sums. Some methods of that proof 
will be used again in this paper. An alternative proof using more extensively 
the method of large sieves appeared as [BG4] . In the case of rational ellipsoids 
the bound 0{s^^) is optimal. For arbitrary ellipsoids Landau [Lai] obtained 
the estimate 0{s~^'^^/^^'^^^), d>l. This result has been extended by Hlawka 
[H] to convex bodies with smooth boundary and strictly positive Gaussian cur- 
vature. Hlawka's estimate has been improved by Kratzel and Nowak ([KNl], 
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[KN2]) to where A = b/{<6d + 2), for d > 8, and A = l2/{Ud + 8), 

for 3 < d < 7. For special ellipsoids a number of particular results is available. 
For example, the error bound 0{s~^) holds for d > 5 and rational Q (see 
Walfisz [Wal], (i > 9, and Landau [La2], d > 5). Jarnik [Jl] proved the same 
bound for diagonal Q with arbitrary (nonzero) real entries. For a discussion 
see the monograph Walfisz [Wa2]. 

Theorem 1.1 is applicable to irrational ellipsoids with arbitrary center for 
d > 9. It extends the bound of order o(s~^) of Jarnik and Walfisz [JW] for 
diagonal irrational Q of dimension d > 5. They showed that o{s^^) is optimal, 
that is, for any function ^ such that ^(s) — > oo, as s ^ oo, there exists an 
irrational diagonal form Q[x\ such that 

limsup s ^(s) A(s, Q, 0) = oo. 

s— >oo 

See Theorem 1.3 for an estimate of the remainder term in (1.2) in terms of 
certain characteristics of trigonometric sums. 

Gaps between values of positive quadratic form.s. Let s,n{s) G Q['Ij'^], 
s < n(s), denote successive values of Q[x]. Davenport and Lewis [DL] con- 
jectured that the distance between successive values of the quadratic form 
Q[x] converges to zero as s — oo, provided that the dimension d > 5 and 
Q is irrational. Theorem 1.1 combined with Theorem 1.1 of [BG4] provides 
a complete solution of this problem for d > 9. Introduce the maximal gap 
d(r; Q,a) = sup{n(s) — s : s >r} between values Q[x — a] in the interval [r, oo). 

Corollary 1.2. Assume that d > 9 and Q[x] is positive definite. If 
the quadratic form is irrational then sup d{T;Q,a) 0, as t ^ oo. // Q 

is rational then su.pd{T;Q,a) < oo. // both Q and a are rational then 

T>0 

inf(n(s) - s) > 0. 

Answering a question of T. Esterman whether gaps must tend to zero 
for large dimensional positive forms, Davenport and Lewis [DL] proved the 
following: Assume that d > dg with some sufficiently large dg. Let e > 0. 
Suppose that y & has a sufficiently large norm |y|oo- Then there exists 
a; G such that 

(1.3) \Q[y + x\-Q[x\\<e. 

Of course (1.3) does not rule out the possibility of arbitrarily large gaps between 
possible clusters of values Q[x\, x G Z''. The result of [DL] was improved by 
Cook and Raghavan [CR] . They obtained the estimate do < 995 and provided 
a lower bound for the number of solutions x G Z'' of the inequality (1.3). See 
the reviews of Lewis [Le] and Margulis [Mar2] . 
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In order to provide bounds concerning gaps between values of positive 
quadratic forms and lattice point approximations for ellipsoids we need addi- 
tional notation. Introduce the trigonometric sum 

(1.4) ipait;s) = {2[^] + l)~^'^ ^exp{itQ[xi + X2 + X3-a]} 

where the sum is taken over all xi,X2,X3 G such that \xj\oo < \/^, for 
j = 1,2,3. Notice that the trigonometric sum (1.4) is normalized so that 
^a{t;s) < (Pai0;s) = 1. 

Theorem 6.1 shows that for irrational Q and any fixed < (5o < T < oo 
the trigonometric sum (pa satisfies 

(1.5) lim sup sup ipa{t;s)=0. 

aeRd So<t<T 

Simple selection arguments show that (1.5) yields that for irrational Q there 
exist sequences T{s) ] oo, T{s) > 1, and do{s) [ such that 

(1.6) lim sup sup (pa{t;s) = 0. 

Sois)<t<T(s) 



The relation (6.5) shows that 

lim sup sup ipa{t;s) = 0, 

for any nondegenerate Q. Hence, the irrationality of Q is equivalent to the 
following condition: there exist T{s) t oo such that 

(1.7) lim 7(5, r(s)) =0, where 7(5, T) = sup sup (pa{t',s). 

aeR'^ s-'^l'^<t<T 

Finally, given d>9, 0<e<l — 8/(i and Q, introduce the quantity 

l-8/<i-£ 



(1.8) p(.)=.i-C+_L_ + (^^(,,T(.))j T-(.), C 



def 1 



d-l 



on which our estimates will depend. Without loss of generality we can assume 
that T{s) in (1.7) and (1.8) are chosen so that 

(1.9) lim p{s) = 0, 

s— +00 

for irrational Q. Indeed, if (1.9) does not hold, we can replace T[s) in (1.7) by 

-il-S/d-e)/(2e)- 



mm 



We shall write A B if there exist a constant depending on d only 
and such that A < caB. 
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Theorem 1.3. Assume that the operator Q is positive, d > 9 and < 
e < 1 — 8/d. Then we have 

(1.10) \Yoh{Es + a)- vol Es \ ^d,e {s + lf/^q'^p{s)s~K 

Theorem 1.1 is an immediate corollary of Theorem 1.3. 

If we choose T{s) = 1 in (1.8) and use j{s,T{s)) < 1, then (1.10) yields 

I voh{Es + a) - vol I <d (s + ^f^^ q'^ s'K 

This shghtly improves the bound (s + 1)^^^ q'^'^'^ s"^ given in [BG4]. 

An inspection of proofs shows that Theorem 1.3 holds for any < d/A. 
Moreover, the main result, Theorems 2.1 and 2.2, can be proved for any real 
2 < p < d/2 (with the expansion in (2.8) defined by the same formula as in 
the case p G N). The assumption p G N is made for technical convenience only. 
Hence, the presence in (1.8) of the term shows that our bound (1.10) can 
not decrease faster than 0(5''/^+'^), 6 > 0. 

Write 

po(s) = supp(r). 

T>S 

Corollary 1.4. Assume that the operator Q is positive and d> 9. Then 



Yolz(^{E^+6+a.)\{Er+a) 



(1.11) \oKE^,,\Er) ^=1 + ^' for r>s>l,6>0, 

where TZ satisfies |7?.|<Cd,£ q^'^/^po{s)/S. In particular, the maximal gap d{s,Q, a) 
satisfies 

(1.12) d{s,Q,a)<^d,eq^'^^^Pois), fors>l. 

The relation (1.11) gives an estimate of the number of values of a posi- 
tive quadratic form in an interval {s,s + S], counting these values according 
to their multiplicities. More precisely, a value, say r = Q[x — a], is counted 
vol^l^; : r = Q[z — a]} times. In the case of irrational Q the approxima- 
tion (1.11) may be applied for intervals of shrinking size S = 5{s) — > as 
s — > 00. The approximation error in (1.11) still satisfies TZ ^ for shrink- 
ing intervals such that S/po{s) — >■ 00 as s — >■ 00. The estimate (1.12) pro- 
vides an upper bound for the maximal gap d{s, Q, a) between values to the 
right of a value .s > 1, for positive Q. In particular, we get, for irrational Q, 
d{s, Q,a) as s ^ 00, uniformly with respect to a. 

The Oppenheim conjecture. Write 

m(g) = inf 1 1 Q[a;] I : x ^ 0, x e Z'^^ 
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Oppenheim ([01], [02]) conjectured that m{Q) = 0, for d > 5 and irrational 
indefinite Q. This conjecture has been extensively studied, see the review of 
Margulis [Mar2]. A stronger version was finally proved by Margulis [Marl]: 
m{Q) = 0, for (i > 3 and irrational indefinite Q. In 1953, A. Oppenheim 
proved in three papers that such a result is equivalent to the following: for 
irrational Q and d > 3, the set Q\^'^] is dense in W^. See the discussion in 
[Mar2, p. 284]. In particular, d(T, (5,0) = 0, for all r, which is impossible for 
positive forms. 

The quantitative version of Oppcnhcim's conjecture was developed by 
Dani and Margulis [DM] and Eskin, Margulis and Mozes [EMM]. Let 
M : ^ [0, oo) be any continuous function such that M(tx) = \t\M{x), 
for alH G M and x G M'^, and such that M{x) = if and only if a; = 0. The 
function M is the Minkowski functional of the set 

(1.13) Q = {xe^'^ : M{x)<l]. 

In particular, the set $7 is a star-shaped closed bounded set with the nonempty 
interior containing zero. For an interval / = (a, 0\ define the set W = {x : 

Q[x — a] G /}. Assuming that d > 5 and that the quadratic form Q[x] is 
irrational and indefinite, Eskin, Margulis and Mozes [EMM] showed that 

(1.14) ^=l + o(l), asi?^oo. 

Furthermore, 

vol {W n {Rn)) =\{P-a) R'^-^ + o{R'^-^), as ^ oo, 

with some A = \{Q,Vt) / 0. Eskin, Margulis and Mozes [EMM] provided as 
well refinements and extensions of (1.14) to lower dimensions. 
Introduce the box 

(1.15) B{r) = {xeW^ : \x\^ < r}. 

Let Co = c{d, e) denote a positive constant. Consider the set 

V = {Q[x-a]: xe B{r/co) } n [-cq r^, cq r^] 

of values of (5[x — a] lying in the interval [— cq r^, cq r^], for x G S(r/co). Define 
the maximal gap between successive values as 

(1.16) d(r) =^ max m.m\ v — u : v > u, v eV\. 
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Theorem 1.5. Let Q[x] be an indefinite quadratic form, d > 9 and 
£ > 0. Assume that the constant cq = c{d, e) is sufficiently small and that 
< cog^^/^r. Then the maximal gap satisfies 

d{r) q^"!'' p(r2), for > Cq ' q'"'l\ 

with p defined by (1.8). 

In Section 2 we shall provide as well a bound (see Theorem 2.6) for the 
remainder term in the quantitative version (1.14) of the Oppenheim conjecture, 
for d>9. This bound is more complicated than the bound of Theorem 1.5 
since it depends on the modulus of continuity of the Minkowski functional of 
the set Q. In this section we shall mention the following rough Corollaries 1.6 
and 1.7 of Theorem 2.6 only. 

Corollary 1.6. Let a quadratic form Q[x] be indefinite and d> 9. Then, 
for any 5 > 0, there exist {sufficiently large) constants C = C{5,q,rt,d) and 
Co = Co{5, q, ft, d) such that 

(1.17) {1-6) vol(T^n(CrO)) < voh{Wn{Crn)) < {l+d)vol{Wn{Crn)) , 
provided that 

(1.18) r>Co, P-a>Co, \a\<r, \a\ + \/3\ < r'^ . 

Corollary 1.6 is applicable to rational and irrational Q. For irrational Q 
the approximations can be improved. 

Corollary 1.7. Assume that the quadratic form Q[x] of dimension d>9 
is irrational and indefinite. Let R = rT^^^. Then there exist T = T{r^) — oo 
such that 

(1.19) ^ f - 1 <^d,m,q g{r) + h{r)/{p - a) ^ 0, as r ^ oo, 

voiiwn{Rn)] 

with some functions 

g{r) = g{r; q,fl, d) and h{r) = h{r;Q,il,d) 

such that g{r),h{r) 0. The convergence in (1.19) holds uniformly in the 
region where \a\ < r and \a\ + < r^. 

We have p{s) ^ 0, as s — > oo (see (1.9)), for irrational Q. Thus, Theorem 
1.5 gives an upper bound for the maximal gap in Oppenheim's conjecture. The 
bound of Theorem 1.5 is constructive in the sense that in simple cases one might 
hope to estimate the quantity p{s) explicitly using Diophantine approximation 
results. In general the estimation of p remains an open question. Corollary 1.7 
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is applicable for shrinking intervals, e.g., for /? — a x ■\/h{r). The bound 
of Theorem 2.6 is much more precise than those of Corollaries 1.6 and 1.7. 
Nevertheless, in order to derive from Theorem 2.6 simple, sharp and precise 
bounds one needs explicit bounds for T, 7(5, T) and the modulus of continuity 
of the functional M. 

Remark 6.2 shows that the results are uniform over compact sets of irra- 
tional matrices Q such that the spectrum of Q is uniformly bounded and is 
uniformly separated from zero. 

The basic steps of the proof consist of: 

(1) the introduction of a general approximation problem for the distribution 
functions of lattice point measures by distribution functions of measures 
which are absolutely continuous with respect to the Lebesgue measure; 
both the elliptic as well as hyperbolic cases are obtained as specializations 
of this general scheme; 

(2) an application to the distribution functions of Fourier-Stieltjes transforms, 

reducing the problem to expansions and integration of Fourier type trans- 
forms of the measures (in particular, of certain trigonometric sums) with 
respect to a one dimensional frequency, say t; 

(3) integration in t using a basic inequality ([BGl], [BG4]; see (3.12) in this 
paper), which leads to bounds depending on maximal values 7 (see (1.7)) 
of the trigonometric sum; 

(4) showing that 7 tends to zero if and only if the quadratic form is irrational. 

Bounds for rates of convergence in the multivariate Central Limit Theorem 
(CLT) for conic sections (respectively, for bivariate degenerate [/-statistics) 
seem to correspond to bounds in the lattice point problems. The "stochastic" 
diameter (standard deviation) of a sum of N random vectors is of order \/iV, 
which corresponds to the size of the box of lattice points. In the elliptic case 
this fact was mentioned by Esseen [Ess], who proved the rate 0(A/"~^+-'^/(^+'^)) 
for balls around the origin and random vectors with identity covariance, a 
result similar to the result of Landau [Lai] . For sums taking values in a lattice 
and special ellipsoids the relation of these error bounds for the lattice point 
problem and the CLT has been made explicit in Yarnold [Y] . 

Esseen's result was extended to convex bodies by Matthes [Mat], a result 
similar to that of Hlawka [H] . 

The bound 0{N-^) in the CLT, for d > 5, of [BG3] for ellipsoids with 
diagonal Q and random vectors with independent components (and with ar- 
bitrary distribution) is comparable to the results of Jarnik [Jl]. The bound 
0{N~^), for d > 9, for arbitrary ellipsoids and random vectors — an analogue 
of the results [BG4] — is obtained in [BG5]. This result is extended to the case 
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of {/-statistics in [BG6]. Proofs of these probabihstic results are considerably 
more involved since one has to deal with a more general class of distributions 
compared to the class of uniform bounded lattice distributions in number the- 
ory. A probabilistic counterpart of the results of the present paper remains to 
be done. 

The paper is organized as follows. In Section 2 we formulate the main 
result, Theorems 2.1 and 2.2, and derive its corollaries and prove the results 
stated in the introduction. Section 3 is devoted to the proof of Theorems 2.1 
and 2.2, using auxiliary results of Sections 4-7. In Section 4 we prove an 
asymptotic expansion for the Fourier-Stieltjes transforms of the distribution 
functions and describe some properties of the terms of the expansion. Section 
5 contains an integration procedure, which allows to integrate trigonometric 
sums satisfying the basic inequality (3.12). In Section 6 wc obtain a criterion 
for Q[x] to be irrational in terms of certain trigonometric sums. In Section 7 
we investigate the terms of the asymptotic expansions in Theorems 2.1 and 
2.2. In Section 8 we obtain auxiliary bounds for the volume of bodies related 
to indefinite quadratic forms. 

We shall use the following notation. By c with or without indices we shall 
denote generic absolute constants. We shall write A <^ B instead of ^ < cB. 
If a constant depends on a parameter, say d, then we write or c(d) and 
use A <Crf B instead of ^ < c^B. By [B] we denote the integer part of a real 
number B. 

We shall write f = [r] + 1/2, for r > 0. Thus r <C r, and for r > 1 the 
reverse inequality holds, r <C r. 

The set of natural numbers is denoted as N = {1, 2, . . . }, the set of integer 

numbers as Z = {0, ±1, ±2, . . . }, and No = {0} U N. 

We write B(r) = ja; G M*^ : |a:;|oo < r\ and |x|oo = max \xA, 

i<j<d 

i<j<d 

The region of integration is specified only in cases when it differs from the 
whole space. Hence, = J and j-^a = /• 
We use the notation 

(1.20) e{t} = exp{it}, i = 

which differs by an inessential factor 2it from often used e{t} = exp{27rit}. 
Since we study forms with arbitrary real coefficients, the convention (1.20) 
suppresses lots of immaterial factors 2tt. 

The Fourier-Stieltjes transforms of functions, say F : M — > M, of bounded 
variation are denoted as 



Fit) = J e{ts}dF{x). 
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Throughout l{^} denotes the indicator function of event A, that is, 
l{ A} = 1 if j4 occurs, and l{ ^} =0 otherwise. 
For s > 0, define the function 

(1.21) M{t; s) = {\t\s)~^ I{ \t\ < } + \t\ l{ \t\ > }. 

For a multi-index a = (ai, . . . ,ad), we write a\ = ail...adl- Partial 
derivatives of functions / : M*^ — C we denote by 

Sometimes we shall use notation related to Frechet derivatives: for a = 
(ai, . . . , ttn), we write 

(1.22) /(l"li)(x)/ir • • • /in" = ^t? • • • dZ-fix + t^h + ---+tnK) 

ti =•••=*„ =0 

Acknowledgment. We would like to thank G. Margulis for drawing our 
attention to the close relation between the quantitative Oppenheim conjecture 
and the lattice point remainder problem and helpful discussions. Furthermore, 
we would like to thank A.Yu. Zaitsev for a careful reading of the manuscript 
and useful comments. 

2. The main result: Proofs of the theorems of the introduction 

For the formulation of the main result, Theorem 2.1, we need some simple 
notions related to measures on R'^. We shall consider signed measures, that 
is, cr-additivc set functions /i : f?^ — > M, where B"^ denotes the cr-algebra of 
Borel subsets of M''. Probability measure (or distribution) is a nonnegative 
and normalized measure (that is, /i(C) > 0, for C e B'^ and fi{R'^) = 1). We 
shall write J f{x) iJ,{dx) for the (Lebesgue) integral over M.^ of a measurable 
function f -.M.^ ^ C with respect to a signed measure fi, and denote as usual 
hy fi * i^iC) = f /^(C* — x) v{dx), for C G B'^, the convolution of the signed 
measures and f. Equivalently, ^x* v is defined as the signed measure such 
that 

(2.1) / fix) * u{dx) = JJ fix + y) iiidx) uidy), 

for any integrable function /. 

Let Px G M, x G Z"', bea system of weights. Using signed measures, 
weighted trigonometric sums, say, 

^{tQ[x]}Px = J Q{tQ[x\] Oidx), e{i;} = exp{if }, 

can be represented as an integral with respect to the signed measure 9 concen- 
trated on the lattice such that Oi^{x}^ = Px, ^oi x ^ U^. 
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The uniform lattice measure lJ^{-;r) concentrated on the lattice points in 
the cube B{r) = {x eW^ : \x\oo < r } is defined by 



CnB{r) 

(2.2) ^iC-r) = —^^^^, for CeB<^. 

In other words, the measure /u(-;r) assigns equal weights fi{{x};r) = {2r)~'^ 
to lattice points in the cube B{r), where f = [r] + 1/2. Notice as well that 
H{-;r) =/i(-;r). 

We define the uniform measure ^'(- ; ^) in -S(r) by 



vol ^CnB(r)^ 



(2-3) -iC-r)=^^-^^, for CeB'^. 

For a number R> write $ = /n(-;i?) and * = z/(-;i?), and introduce 
the measures 

(2.4) n = ^*fi*''{-;r), u = * u*'' {■;¥), € N. 

The distribution function, say G, of a quadratic form (^[a; — a] with respect 
to a signed measure, say A, on is defined as 

(2.5) G{s) = X{x eR'^ : Q[x - a] < s} = J l{Q[x - a] < s} X{dx), 

where l{^} denotes the indicator function of event A. The function 
G : M ^ R is right continuous and satisfies G(-oo) = 0, G(oo) = X{W^). If 
A is a probability measure (i.e., nonnegative and normalized) then we have in 
addition: G : M ^ [0, 1] is nondecreasing and G(oo) = 1. 

We shall obtain an asymptotic expansion of the distribution function, 
say F, of Q[x — a] with respect to the measure /x defined by (2.4). The first 
term of this expansion will be the distribution function, say Fq, of Q[x — a] 
with respect to the measure u defined by (2.4). Other terms of this expansion 
will be distribution functions Fj, j G 2N, of certain signed measures related 
to the measure u (or, in other words, to certain Lebesgue type volumes). A 
description of Fj will be given after Theorem 2.2. 

Introduce the function (cf. (1.4)) 

(2.6) <fa{t; r^) = \j e{tQ[x - a] } ii*^dx; r) 
and, for a number T > 1, define (cf. (1.6) and (1.7)) 

(2.7) 7(r2,T)1^%up sup Mt;r^)- 

' —^<t<T 



Our main result is the following theorem. 
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Theorem 2.1. Assume that 

d>9, pen, 2<p<d/2, k>2p + 2, < r < R, T > 1. 
Then the distribution function F allows the following asymptotic expansion 

(2.8) F{s) = Fo{s)+ E Fj{s) + n 

je2N, j<p 

with a remainder term TZ satisfying 

(2.9) ^ + 1^ (l + ^) V+'^/^ + 7^-«/^-^(r^T) T^^, 
for any £ > 0. 

Notice, that the estimate (2.9) is uniform in s. 

The measure $ (or its support B{R) nZ'^) represents the main box of size 
R from which lattice points arc taken. The convolution of $ with ix*^{-;r) is 
a somewhat smoother lattice measure than Note though that the weights 
assigned by ^ to the lattice points near the boundary of the box B{R) become 
smaller when the points approach the boundary of the box B{R + kr). The 
weights assigned to lattice points in B{R— kr) remain unchanged. Later on 
we shall choose the size r of the smoothing measure lJi{-',r) smaller in compar- 
ison with R, that is, we shall assume that R > ckr with a sufficiently large 
constant c. This smoothing near the boundary simplifies the derivation of ap- 
proximations and helps to avoid extra logarithmic factors in the estimates of 
errors. The corresponding measure v is the continuous counterpart of /j, with 
the dominating counting measure on Z*^ replaced by the Lebesgue measure on 
W^. Theorem 2.1 allows a generalization. The measure $ can be replaced by 
an arbitrary uniform lattice measure with support in a cube of size R, see 
Theorem 2.2 below. Theorem 2.1 is a partial case of Theorem 2.2. We shall 
prove Theorem 2.2 in Section 3. In order to formulate that result, we extend 
our notation. 

We shall denote tt = i^(-;l/2). The measure tt has the density = 
l{ \x\oo ^1/2} with respect to the Lebesgue measure in M*^, so that 7:(dx) = 
l{\x\oo < 1/2} dx. The measure ui-: r) has the density (2r)-'^l{| < r}. 
Notice as well that z^( •; f) = tt *//(•; f). 

Henceforth $ will denote a probability measure on M*^ such that ^{A) = 1, 
for some subset A(ZB{R)nZ'^ and 

(2.10) ^{{x}) = 1/card A, for all x e A. 

We do not impose restrictions on the structure of A except that A C B{R){^'L'^ 
and ^ 7^ 0. Write * = $ * tt. It is easy to see that ^ has the density 

(2-11) f = ^P/{l--^loo<V2}. 

y&A 
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(2.12) D^^^'ig ^ = ^ J M-,r) 



We define measures /j, and v as in (2.4), and denote distribution functions of 
Q[x — a] with respect to /x and u as F and Fq respectively. Notice that the 
measure u has the density 

[I 

dx dx \ dx 

where f * g denotes the convolution of functions / and g, 

f * 9{x) = J f{x- y) g{y) dy. 

Using the Fourier transform, we can easily verify that the density D admits 
continuous bounded partial derivatives \d'^D{x)\ <^d,k r^*^"!*^!!, for |q;|oo < 
k — 2 (see Lemma 7.1). 

Theorem 2.2. Theorem 2.1 holds with $ and defined by (2.10) and 
(2.11) respectively. 

Let us now define the functions Fj, for j G 2N. Let rj = (r/i, . . . , 77^) 
denote a multi-index with entries r)i, . . . ,r)m G N. Write ** for the sum 

V- \ri\i=j 

which extends over all possible representations of the even number j as a sum 
j = rji + ■ ■ ■ + rjm of even 771, ... , rjm > 2, for all possible m > 1. For example, 
for j = 6, we have 6 = 6, 6 = 4 + 2, 6 = 2 + 4 and 6 = 2 + 2 + 2. Introduce the 
functions 

(2.13) Dj{x)= ^i.(^) 

V- \v\l=3 



with 

m 

f- [■■■fD(^Hxwr...u't 



(2.14) D^.^(x)= /•••/DW(xX...nr X\T,<^^^){dui), 

where the density D is defined by (2.12), and where we use the notation (1.22) 
for the Frechet derivatives. For example, we have 

D2(X) = -|/ D"{x)u^TT<^+^\du), 

and Di{x) = D^{x) + .0422(3^) with 

D44(x) = - ^ / D^*\x)u^^*^^+^\du), 

Da22{x) = I // D^^\x)ululT:<^+^\dui)TT<''+^\du2). 



Let Vj denote the signed measure on with density Dj. We define the 
function Fj, for j G 2N, as the distribution function of Q\x — a] with respect 
to the signed measure Uj] that is, 

(2.15) Fj{s) = Vj{x : Q[x - a] < s] = l{Q[x - a] < s] Dj{x) dx. 
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The function Fj : M ^ M is a function of bounded variation, Fj (—00) = 
Fj {00) = and 

(2.16) sup| F,{s) I ^ (l + q'^"/', for j < d/2; 

see Lemma 7.4. 

In the elliptic case the choice of $ is immaterial as long as the support of 
<!• contains a sufficiently massive box of lattice points. Thus we shall simply 
choose $ and ^' as in (2.4). The same choice of $ is appropriate for the 
estimation of maximal gaps (cf. Theorem 1.5) in the hyperbolic case. The 
choice of a general as possible $ is appropriate for proving refinements of 
(1.14). We shall restrict ourselves to the following special $ generated by a 
star-shaped closed bounded set O. (see (1.13)) whose nonempty interior contains 
zero. Define 

and let in accordance with (2.10) the set A be given hy A = (Rft) H U^. The 
measure ^ is again defined by (2.11). In order to guarantee that |x G Z'' : 

$({a:}) > o| C B{R), we shall assume throughout that CI C B{1); this is not 

a restriction of generality. Hence, for the Minkowski functional of the set J7 we 
have 

(2.18) |a;|oo < |M(x)| < m|x|oo, for all x G M'^, 

with some m > 1. The inequalities (2.18) are equivalent to B{l/m) C 
CB(1). 

The modulus of continuity 

(2.19) uj{5)= sup \M{x + y)- M{x)\ 

\y\oa<S, |a;|oo=i 

of M satisfies \im.u{5) = 0. For $ and * in (2.4) we have Q, = B{1), M{x) = 
5— »0 

|a;|oo and uj{6) = S. 

def 

Let {dQ)a = dQ + B{a) be a tj-neighborhood of the boundary dQ of fi. 
Then, introducing the weight po = 1/ vol^(i?il), writing for a while a = kr/R 
and assuming that <I> is defined by (2.17), we have 

(2.20) ^(C) = po voh C, ioi ccR{n\ {dn)^ ) , 
H{C) = 0, for C C M*^ \ (Rn^). 
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Notice that po = {2R)~'^ for $ defined after (2.4). In order to prove (2.20), 
it suffices to consider the case when the set C is a one point set, and to use 
elementary properties of convolutions. Similarly, for measurable C C M*^, we 
have 

(2.21) v{C)=poJdx, for C C R{n\{dn)a), 

c 

< v{C) <po J dx, for C C R"^, 

v{C) = 0, for CcR'^X {Rn^), 

where now a = {kr + l)/R. Using (2.19), it is easy to see that 
(2.22) 

R{dn)a- C {xeW^ : 1 - uj{a) < M{x/R) < l + uj{a)}, for any a > 0. 

We conclude the section by deriving ah results of the introduction as 
corollaries of Theorem 2.1; the refinement of (1.14), Theorem 2.6, is implied 
by Theorem 2.2. 

The elliptic case. Let us start with the following corollary of Theorem 2.1. 

Corollary 2.3. Assume that the operator Q is positive and T > 1. 
Then we have 
(2.23) 

\vohiE,+a)-volEs\«,^e {s+lf'q'^^'/' ( J_ + _L (,, t) 

for d>9, 2 < p < d/2, p G N and any e > 0. The quantity 7(5, T) is defined 
in (1.7) (c/. (2.6) and (2.7)). 

Proof. The bound (2.23) obviously holds for s < 1. Therefore proving 

(2.23) we shall assume that s > 1. 

We assume as well that |o|oo < 1- This assumption does not restrict 
generality since 

(2.24) yoh{Es + a) = voh{Es + o - m), for any m G Z*^, 

and in (2.23) we can replace a by a — m with some m G Z*^ such that 
\a — m|oo < 1- 

Choose the measure $ as in (2.4), and 

k = 2p + 2, r = Vi, R = 2kr. 

Obviously R <jk -y/s + 1, for s > 1. Therefore the bound (2.9) of Theorem 2.1 
implies (2.23) provided that we verify that our choices yield 

(2.25) 

F{s) = {2R)-'^ volz(^s + a), Fq{s) = {2R)-'^ volE,, Fj{s) = 0, for j 7^ 0. 
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Let US prove the first equality in (2.25). The elhpsoid Ei is contained 
in the unit ball, that is, Ei C {\x\ < 1} C -B(l), since the modulus of the 
minimal eigenvalue (^g is 1 . Therefore we have Eg C y/s ) . Due to our choice 
of R, r and A; > 6, we have R — kr > 6y/s. Thus, the inequality |a|oo < 1, 
the relations Eg + a C B{1 + y/s) and F{s) = ji{Es + a) together with (2.20) 
imply the first equaUty in (2.25). 

Let us prove the second equality in (2.25). Using (2.12) and (2.21) we 
see that the density D is equal to zero outside the set i?(i? + A;r + l), and 
D{x) = {2R)-'^, for X G B{R -kr-1). The ellipsoid E^ + a is a subset of 
B{R — kr — 1), that yields the second equality in (2.25). 

For the proof of Fj{s) = notice that the derivatives of D vanish in 
B{R — kr — 1), hence in the ellipsoid Eg + a as well. □ 

Proof of Theorem 1.3. This theorem is implied by Corollary 2.3. Indeed, 

the estimate (1.10) is obvious for s < 1. For s > 1, the estimate (1.10) is 
implied by (2.23) estimating qP < q'^^'^, choosing p = and T = T{s) as in 
the condition of Theorem 1.3. □ 

Proof of Corollary 1.4. We have to prove (1.11) and (1.12). The proof of 
(1.12) reduces to proving that 

\o\i{Er+8 + a) - vohiEr + a) > 0, 

for 5 > c{d,e)q^'^^'^ Po{s) with a sufficiently large constant c{d,e). Using (1.11) 
it suffices to verify that \Tl\ < 1/2, which is obviously fulfilled. 

Let us prove (1.11). Consider an interval (r, r + 6] with r > s > 1. We 
shall apply the bound of Theorem 1.3 which for s > 1 yields 



(2.26) 

We get 
(2.27) 



volz{Es + a) - vol Eg 



«d,e q'^S-'+'^l^ 



Pois). 



Er+s + a)\{Er + a))- vol{Er+s \ Er) 

^d,e (r + 5f^-^ {po{T + 6)+ p,{t)). 

The estimate (2.27) implies (1.11). Just note that Po{t) < po{s), for t > s, 
divide both sides of (2.27) by vo\(^Et-^s \ Br) and use 

Yol{Er+s \Er) = {{T + - t"/'') wo\E^, 
(^+<5)d/2_^<i/2 du »d (5(r+,5/2)-i+'^/2 ,5(t+5)-i+<^/2, 

T+5/2 



\o\Ei = vol{x G M'' : Q[x] < l} > vol{a; G M'* : |x| < l/^} = caq 



-d/2 



□ 
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Proof of Corollary 1.2. It suffices to use (1.9) and (1.12). □ 

Proof of Theorem 1.1. Assuming the irrationality of Q, the bound o{s~^) 
is implied by Theorem 1.3 and (1.9) since vol -Eg g~'^/^s'^/^. 

The bound o{s-^) in (1.2) implies d(r,Q,0) 0, as T — ^ oo, which is 
impossible for rational Q. □ 

The hyperbolic case. For an interval / = (a, /3] C M we write 

(2.28) F{I) = Fi/3) - F{a), Fj{I) = Fj{l3) - Fj{a). 

Notice that F{I) = F{s) in the case / = (— oo,s]. Theorem 2.2 has the 
following obvious corollary. 

Corollary 2.4. Under the conditions of Theorem 2.2 we have 

(2.29) F{I) = Fo{I)+ E Fj{I)+n 

with remainder term TZ which satisfies (2.9). 

Lemma 2.5. Let Q[x] be an indefinite quadratic form of dimension d >9. 
Letp, k ande satisfy the conditions of Theorem 2.1. A ssume that ikf(x) — l^^loo 
and Q = B{1). Let ci = ci{d,e) denote a sufficiently small positive constant. 
Finally, assume that 

(2.30) R>r>^, a,P e[-ciR^,ciR^], ^ < -y, ^/q\a\<ClR. 

Then 
(2.31) 

Proof. The result follows from Corollary 2.4 dividing (2.29) by Fq{L) and 
using the estimates 

(2.32) E I Fj{I) I <^k,d iP - a) r-^-'i R''-\ 
je2N, j<p 

(2.33) Foil) >A:,d (/? - a) q"^''' R-\ 

Let us prove (2.32). Using (2.15), (2.28), (2.30), (7.1), applying the esti- 
mate (8.9) of Lemma 8.2 with 

M(x) = |x|oo, R = R + kr, and m = A = 1, 



no 
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using the bound |ao| < we obtain 

(2.34) 

\ ^k,dr-^-'^ J I{\x\oo <R + kr}l{Q[x-a] e l} dx 

In the proof of (2.34) the condition i? > r > 1/ci ahowed us to replace R and r 
by R and r respectively. Summation in j, 2 < j < p, of the inequalities (2.34) 
yields (2.32). 

Let us prove (2.33). Using (2.12), (2.21), the lower bound (8.10) of Lemma 
8.2 and conditions (2.30), we have 

Fo(7)»d i?-'^/l{|x|oo 

<R-kf}l{Q[x-a] el}dx:^d{P-a)q-'^/^R''^. □ 



Proof of Theorem 1.5. We shall apply Lemma 2.5 choosing T = T{r'^), 
R = rk/ci, the maximal p and minimal k such that the conditions of Theorem 
2.1 are fulfilled. In this particular case we can rewrite (2.31) as 



(2.35) 



MI) 



1 



qd+p 

13 -a 



p{r'^ 



In order to estimate the maximal gap, it suffices to show that any interval 
/ = (a,/3] contains at least one value of the quadratic form (i.e., F{I) > 0) 
provided that a and j3 satisfy /3 — a ~>d,e Q^'^^ pii"^)- The inequality F{I) > 
holds if the right-hand side of (2.35) is bounded from above by a sufficiently 
small constant which can depend on d and e. □ 

Next we formulate and prove some refinements of (1.14). Recall that 

the set satisfies B{l/m) C Q C B{1) (see (2.18)), and that uj denotes the 
modulus of continuity of the Minkowski functional M of the set (see (1.13) 
and (2.19)). Write 

(2.36) W={xeR'^: Q[x 

A* - ^ 



(2.37) 

and 
(2.38) 



(a,/3], 



vol 



1 



eo = r/R, ei = ^\a\/R, 

e3 = w(eo/c2), = uj{ei^/c2), 

where C2 = C2{d,m) denotes a positive constant. 



S2 = {\a\ + \(3\)/R\ 
£5 = ^^{£2^/q/c2), 
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Theorem 2.6. Assume that the form Q[x] is indefinite and d > 9. Let 
r > l/c2, < £ < 1 - 8/d, T>\ and 

(2.39) Ej < C2, for j = 0, 1, 2, 3, ej < C2 q-'/\ for j = 4, 5. 
Then we have 

(2.40) A* <^d,e,m q"-^ (l + ^ ) (£3 + £4 + £5) 

^ 0-oc \ elT ^ r--^+<i/2 ef+^ > e'i ) 

provided that the constant C2 is sufficiently small. 

Proof of Corollary 1.6. Let us apply Theorem 2.6 choosing 02 = 
02(6, q, d, m) sufficiently small depending on S and q as well. Choose R = Cr. 
Then we have eq = and the last two inequalities in (1.18) guarantee condi- 
tions (2.39). In particular, we have ej < C2, for j = 3, 4, 5. Hence, we can apply 
the bound (2.40). Choosing T = 1, e = (1 — 8/d)/2 and estimating 7 < 1, we 
obtain 

(2.41) A* C2 (1 + r-^C) + (C^ + C''+^)/{P - a), 

for r > 1. The estimates (1.17) follow if the right-hand side of (2.41) is bounded 
from above by S. But this holds in view of the first two inequalities in (1.18) 
and our choices of C, Cq and C2- □ 

Proof of Corollary 1.7. During the proof we shall write T = T{r^) and 
7 = 7(r^,T). Choose e = (1 — 8/d)/2. Since 

R = rT^/"^ with T —>■ 00, we 
have eo = T~^/^ and the conditions (2.39) are fulfilled for sufficiently large r. 
Moreover, ej 0, for j = 3,4,5. Hence, the bound (2.40) yields (1.19) with 

5(r) = (l + r-2r'^/4)(e3 + e4 + £5), 

^ 2^-1/2 _^ ^-1 j.(d+2)/4 _^ ^(l-8/d)/2 rpl-4/d_ 

If g{r) ^ or h{r) 0, we can choose (if necessary) T = T{r'^) — 00 growing 
somewhat slower such that g{r) — ^ or h{r) — ^ (cf. a similar redefinition of 
T{s) in the case of (1.9)). □ 

For the proof of Theorem 2.6 we shall need the following Lemma 2.7, 
which allows us to estimate A* using Theorem 2.2. Write 

(2.42) A° = Yoh{w n (Rn)) -Yoi{wn{Rn)). 

By = $ * A**'^( • ; r) and un = ^ * ; ^)) where = $ * tt, we shall denote 
the measures /U and v, emphasizing the dependence on the parameter R which 
enters in the definition (2.17) of 
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Lemma 2.7. Let ao = {kr + l)/R. Assume that Lo{2ao) < 1. Write 

(2.43) si = R/{l + Loi2ao)), S2 = 1 - a;(2ao)) , P2 = V volz(s2^^). 
Then we have 

(2.44) \A°\<p2^ max \As\ + v, 

S=Sl,S2 

where 

V = vol r\ {si{l - uj{2ao)) < M{x) < 52(1 + w(2o-o)) } 

and 

(2.45) As = Jl{xeW}ns{dx)- Jl{xeW}i^s{dx). 
IJu{2uq) < 1/4 then 

(2.46) V < vol I^V^n {1 - 3cc;(2c7o) <M{x/R) < l + 3u;(2ao)} 
Proof. We omit the elementary proof of (2.46). 

Using si = R/{l + u;{2ao)) > R/2 and the fact that a; is a nondecreasing 
function, it is easy to see that 
(2.47) 

si(l +uj(ai)) < R < 82(1 - io(cr2)), with ai = + ^ (72 = ^'""'"^ , 

Let us prove (2.44). Assume first that A° > 0. Applying (2.20)-(2.22) to 

fig and i^s and using (2.47), we have 

(2.48) Ais2(C) =P2 vohC, for C C RQ, 

lys^iC) <P2^ol(^Cn{s2{l + u;{2ao))n)y for any C. 
Using (2.48), we have 
(2.49) 

Yoh{wn{Rn)) = fis,,{w n (Rn)) <p^^ Ji{x ew} ns^idx), 

fl{x e W}us2{dx) < vol{Wn{Rn)) +v, 

proving the lemma in the case A° > 0. 

Assume now that A° < 0. Write pi = l/volz(siri). Relations (2.20)- 
(2.22) together with (2.47) yield 



Yoi{wn{Rn)) < vol i^w n {si{i - oj{2ao))n) j +v, 

vol{W n {slil - ij{2ao))n)) <p^^ fl{xeW}i^sMx), 

voh{wn{Rn)) >pi^ ji{x^w]iisAdx), 

and using p2 < pi we obtain (2.44). □ 
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Proof of Theorem 2.6. The result follows from the bound 

(2.50) A* <^d,e,m,p,k q"-' (l + ^ ^) (^(2ao) + £4 + £5) 

choosing the maximal p < d/2, /c = 2p + 2 and using the notation (2.38). 
Recall, that we assume that the constant C2 in (2.38) is sufficiently small. 
In particular, this assumption guarantees that a;(2cro) is as small as will be 
required in the auxiliary lemmas used below. 

Let us prove (2.50). Dividing the bound (2.44) of Lemma 2.7 by vi =^ 
vol ( n (i? O) ) , we obtain 

|A*| < ^— max lAJ + — . 

P2V\ S=S1,S2 ^1 

Hence, in order to prove (2.50) it suffices to show that 

(2.51) p^^ =Yo\z{s2n)<^dR\ 

(2.52) :$>d,m iP - a) q-''!^ i?'^-^, 

(2.53) V <^d,m {P - a) {u{2ao) + £4 + £5) q^"^'^^^^ B!^'^, 
and, for s = si,S2, 

(2.54) |A,| < lA^^il + |A,,2| 
with Ag^i and As^2 defined below by (2.57) such that 

(2-55) |A,,| ^ + ^/+^/^ + V-«/^-(^^T)T^^. 

(2.56) |A,,2| «d,m (/? - a) {u{2ao) + £4 + £5) 9^'^"'^/' r'^ (R/r) 



d-2 



To prove (2.51) it suffices to notice that S2 < 2R since O C B{1) and we 
assume that uj{2ao) is sufficiently small. 

The estimate (2.52) follows from (1.13) and (8.10) of Lemma 8.2 choos- 
ing A = 1 and using the estimate |oo|/-R < ^yq\a\/R = £1 < l/(2m) which is 
guaranteed by the assumption that the constant C2 is sufficiently small. 

The bound (2.53) follows from (2.46) and Lemma 8.3 with S = 3u}{2ao) < 
1/4 which is fulfilled since C2 is small. Applying Lemma 3.8 we use the esti- 
mate £1^0 < ^1. 

Let us prove (2.54). Let F(I;i?) and Fj{I;R) denote the functions F[I) 
and Fj{I) defined by (2.28) with the underlying parameter R which enters into 
the definitions of the measures ijl = jir and v = vr. Then (2.54) holds with 

(2.57) A,,i = F(J;s)-Fo(7;s)-A,,2, A,,2 = E Fjil-.s). 

j62N, 3<p 
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For the estimation of A^^i we shall apply Theorem 2.2. Our choice of the con- 
stant C2 yields oj{2ao) < 1/2. Therefore s = si, S2 is asymptotically equivalent 
to R. The functions ,s F{I] s) and s — > s) are differences of the corre- 
sponding distribution functions. Furthermore, \a\ <S R. Hence, Theorem 2.2 
yields (2.55). 

It remains to prove (2.56). Let us estimate Fj{r, s). Using s >i R, r >i f 
and (7.4), we have 

\Fj{I;s)\ <^d,p V2r-^-'^, 

where 

V2 = vol n I a; gM'^: \M{x/s)-1\ <u{ckr/R) 

By Lemma 8.3 and a suitable choice of C2, the volume V2 allows the same upper 
bound as the volume v in (2.53) since s x i?. Hence, summing these bounds 
over j, we get (2.56). □ 




3. Proof of Theorems 2.1 and 2.2 

We shall use the following approximate (see, e.g., relation (8.4) in [BG4]) 
and precise (see, e.g., Chung [Ch]) formulas for the Fouricr-Stieltjes inversion. 
For any T > and any distribution function F of a normalized nonnegative 
measure with the Fourier-Stieltjes transform 

F{t) = Je{tx}dF{x), teR, 

we have 

(3.1) F{x) = 1 + ^ V.P. J^e{-xt}F{t) f +n 

with remainder term TZ such that 

1^1 < ^ J\F{t)\dt. 

-T 

Here V. P. J" f{t)dt = lim J f{t) dt denotes the Principal Value of the inte- 

gral. Furthermore, for any function F : M — R of bounded variation such that 
F{-oo) = and 2F{x) = F{x+) + F{x-), for all a; e M, we have 

(3.2) F(x)=iF(oo)+^ lim V.P. / e{-xt}F(t)f. 

The formula is well-known for distribution functions. To functions of bounded 
variation it extends by linearity arguments. 
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Theorem 2.1 is implied by Theorem 2.2 and we have to prove Theorem 
2.2 only. 

The expansion (2.8) yields 



(3.3) 



I7^l 



je2No, j<p 



Let us prove Theorem 2.2 assuming that r < 1. Using (3.3), Lemma 7.4 
to bound \Fj\, and the obvious estimates \F\ < 1 and \Fq\ < 1, we obtain 

(3.4) |7^|«.l+ E ^(i+^)V^'^/^«.^(i + ^)V+'^/^ 

l<j<p 

since q > 1, 1 < R/r, 1/r > 1 and j < p < d/2. The estimate (3.4) implies 
the theorem in the case r < 1. Therefore in the remaining part of the proof 
we can and shall assume that r > 1. 

Using the representation (3.3), representing F by the approximate Fourier- 
Stieltjes inversion (3.1) and Fj by the inversion formula (3.2), splitting the 
intervals of integration, and using the triangle inequality and the obvious esti- 
mate 



l/r<\t\<T l/r<|t|<T ' ' 



we obtain 
(3.5) 

with 



< /i + /2 + /a + E Ij+A 
je2No, j<p 



I \Fit)- Yl ^^(i) 



\t\r<l 



je2No, j<p 



dt 



l2= ^ 



I F{t) I dt, 



\t\r<l 



dt 



\t\r>l, \t\<T 



-I 



j e 2No, j <p. 



\t\r>l 

The estimate (3.5) shows that in order to prove the theorem it suffices to prove 

that 

RP / Inl \P 



(3.6) 
(3.7) 



j e 2No, j <p, 



h^d 
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(3.8) h <^,u 7^~^^'^"'7(^^ T)T' for any e > 0. 
Let us estimate I3. Changing variables, we have 

F{t) = Je{tQ[x-a]} n{dx). 

Splitting 

(3.9) ^ = ^*^{.;r)*x with x = 
we obtain 

(3.10) is<r„ r.^^'snp J Mt;r')^, 

aeRrf r|t|>l, |t|<T ' ' 

where ipa is given by 

(3.11) Mt;r'') = \je{tQ[x-a]}tJ,*^{dx;r) 

(cf. (2.6)). The function satisfies the following inequaUties (see Theorem 
5.1 in [BG4]) 

(3.12) ipait;r^)Mt + r;r^) q"'^ M^'^r-y), 

(3.13) v'a(T;r2) «rf//2A^'^/2(^;r2), i^^g^ 

involving the function M.{t\r'^) introduced in (1.21). The inequahty (3.12) 
allows us to apply Theorem 5.1 of the present paper. Choosing in this theorem 

A = Crf//^ x = d/2, s = r^, a = -l, 

and using Inx x*^, for a; > 1 and £ > 0, we obtain 

/ T(r•^T) .1-8/d-e d/2 

(3.14) (1 + lnT)^. 

Estimating in (3.14) 1 + InT for T > 1, and g > 1, and using (3.10), 

we derive the desired bound (3.8) for I3. 

Let us estimate I2 ■ Similarly to the estimation of we obtain 

(3.15) /2< I'2= sup / Mt;r^)dt. 

oeKd r\t\<l 

The bound (3.13) for and the definition (1.21) of the function M{t;r'^) 
together with the inequality (/^a < 1 yield 

(3.16) Mt;r^) <d g'^/' min{ 1; (r^ |i|)-'^/2 }, for r\t\ < 1. 

Hence, we have 

/2<d9''/'/mWl; (r2|t|)-'^/2}dt= 4^ /min{l; \t\-''/^} dt 4^, 
which together with (3.15) yields the bound (3.7) for I2. 



LATTICE POINTS 



1001 



Let us prove (3.6) for Ii. Applying the bound (4.10) to \TZ\ defined by 

(3.3) , bounding by (pa the function in (4.10) and using (3.16), we obtain 

with 

(3.17) I[= f ((|t|r2)^(^) + (|t|r^)^)min{l; (Itlr^)-'^/^}^. 

r\t\<l ^ ' 

By change of variables ir^ = r we get 

oo 

(3.18) < /(r^(f) + rf ) min{ 1; r-^l^ } — 1, 

^ 

since < 6'(p) < p < d/2, see (4.7). The bounds (3.17) and (3.18) yield (3.6) 
for l\. 

Let us prove (3.6) for /j+4. Using (7.13) we have 

^.>4«.-5,^(i + ^)%^-+^/^; 

an application of the inequalities 

r > 1, Rjr > 1, 3 <v< d/2, q>l 
implies (3.6) for /j+4, thus concluding the proof of Theorem 2.2. 

4. An expansion for the Fourier-Stieltjes transform F 

By change of variables we can write the Fourier-Stieltjes transforms of the 
distribution functions F and Fq as 

(4.1) F{t) = Je{tQ[x-a]]ii{dx), %{t) = J e{tQ[x - a]] v{dx). 

Similarly, for the Fourier-Stieltjes transforms of the functions of bounded vari- 
ation Fj (see (2.13)-(2.15)) we have 

(4.2) F^{t)= E** F^vit), 

n- \n\i=3 

where the sum E** is defined as in (2.13), and 

■n- Hi=3 

(4.3) 

Fjr,{t)= J---Je{tQ[x-a]}D(^){x)uf...vPr;r dx H 7r*M(d^x,). 

1=1 

Introducing the notation 

(4.4) g{x) = exp{/i(x)}, h{x) = itQ[x — a], 
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integrating by parts and using D(x) dx = u{dx), we have 

(4.5) F{t) = J g{x) n{dx), Fo{t) = J g{x) u{dx) 
and 

(4.6) F,„(i) = J--Jgi^\x)uf...u'^u{dx) U 7r*M(dnO. 

1=1 

In the proof of (4.6) we used that the function D has compact support and the 
derivatives ff^D are continuous functions, for |q;|oo < k — 2. 
Write 

(4.7) e{s) = s/2, for even s, and e{s) = {s + l)/2, for odd s, 
and 

(4.8) V(i) = sup / e{ tQ[x] + {a, x) } {dx; r) 

(4.9) Vo(t) = sup je{tQ[x\ + {a, x) } {dx; r) 



Lemma 4.1. Let 2 < p < k — 1 and q > 1- Then, for R> r > 1, we have 



(4.10) 



F{t)-Fo{t)- E m) 



<...^{i+^y{mr'f''Hinrr) 



and, for j e2No, j <p, 

(4.11) \m\ (i+ K)^- ((1^1,2)^0) + 

Proof. Let 5 : M'^ ^ C denote a sufficiently smooth complex valued func- 
tion. Assume that x,ui, . . . ,Up € W^. We shall use the following decomposition 

(4.12) g{x)=g{x + ui) + gi + ---+gp, 
where 

(4.13) gj= E* c{r])g^^\x + um+i)uf ...u'^, for 1 < j < 

(4.14) gp= E* Cp(r?)/(l-r)''™<7(^)(a; + r«^X...«rdT, 

??: |??|i=p 



and 

(4.15) c(r?) 



-1)- 



{-ir 



r?i! . . . rim-i\irim - 1)! 



Here the sum E * is taken over all representations of j as a sum j = r]i + - • • + 

v- l»?|l=J 

??m) ^ j, of integers r/i, . . . , € N. For example, in the case j = 3 we have 
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the following four representations: 3 = 3, 3 = 2 + 1, 3 = 1 + 2, 3 = 1 + 1 + 1. 
In particular, ii p = 1 then 

1 

gi = - /(I - r) g'{x + TUi)ui dr, 


and if p = 2 then 

51 = -g'ix + U2)UI, 

1 1 

52 = J ^"^^ ■'"^2)'"i'"2 rfr — y (1 — t)^ g"{x + Tni)ui dr. 



In order to prove (4.12) it suffices to itcrativcly apply Taylor expansions. 
In the first step we use the Taylor expansion of the function g: 
(4.16) 

g{x) = g{x + Ki) - E -^g^'^'K^X - /(I - rygiP){x + t«i)< dr. 

??i=i '■'^ ''• 

In the second step we apply expansions of type (4.16) to the functions x 
g^^'^\x)u^ , for 1 < r/i < p, using U2 instead of ui. After p such steps we arrive 
at (4.12). 

For the derivatives of the composite function g{x) (see (4.4)), we shall use 
the following formula, which follows from a general formula (see Theorem 2.5 
in Averbuh and Smoljanov [AS]). Let . . . , t;^ G M'^. l^ct A = {Ai, . . . , A. } 
denote a partition of the set { ui, . . . , } = Ai U • • • U A^, into a disjoint sub- 
sets Aj such that 1 < card Aj < 2, for all j. Notice that 9{s) < a < s, where 
6{s) is defined by (4.7). For a subset A C {vi, . . . ,Vs}, say A = {vi^, ... ,v^}, 

introduce the derivative dAh{x) =^ h^^\x)vi^ ■ ■ - v^. Then 

(4.17) g^'\x)vi ...vs = g{x) E OaMx) ■ ■ ■ dAM^), 

A 

where the sum E extends over all partitions A with properties specified above. 
A 

One can easily prove (4.17) using h"'{x) = and induction in s. 

The following identities 

(4.18) u{-;r) = fi{-;r)*7r, u = fi * 7r*^''+'^\ 

are obvious (see (2.2), (2.3) and (2.4) for the definitions of measures which 
appear in (4.18)). For example, for any integrable function : M*^ — M, we 
have 

/ u{x)i'{dx;f) = E {2f)~^ J u{x+y)Tr{dx) = JJ u{x+y) Tr{dx) iJ,{dy;r), 

(r) 
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proving the first identity in (4.18). 

Integrating both sides of the identity (4.12) with respect to the measures 

/v,(dT), 7r*(^'+i)(dni), 7r*('^+i)(dnp) 

and using (4.5), (4.6), (4.13), (4.14), (4.18), we obtain 

(4.19) F{t) = hit) + hit) + ■■■ + /p_i(t) + fp{t), 
with F and Fq defined by (4.5), 

m 

(4.20) fj{t)= E* civ) J---Jg^^Hx)u'l\..u'^u{dx) U^*^'''''Hdus), 

r?: |r;|i=j s=l 
for 1 < j < p, and 

(4.21) fpit) = E * cpiv) /(I - r)^- fpit; V) dT, 

rj: \r)\i=p 

A f 

fpit; V) = 9^^ ix + TUm)u\' ...ul^ flidx) n 7r*(*^+i) idUl). 

1=1 

The sums in (4.20) and (4.21) are the same as in (4.13) and (4.14) respectively. 
However, notice that fj = 0, for odd j < p, since the measure tt is symmetric. 
Moreover, fj = Fj, for even j < p, since all terms in the sum (4.20) vanish 
unless all rji, ... ,r]jn are even, due to the symmetry of vr. Hence, (4.19) yields 

(4.22) Fit)-Foit)- E Fjit) =\fpit)\. 

je2N, j<p 

Now we can return to the proof of (4.10). The equality (4.22) shows that it 
suffices to verify that | /^(t) | is bounded from above by the right-hand side of 
(4.10). Using (4.21), we have to verify that any of the suprema sup I /p(t; 77) I 

0<r<l 

is bounded by the right-hand side of (4.10), for all allowable ryi, . . . ,77^. Define 
vi,...,Vp repeating ui rji times, followed by U2 % times, etc. For example, 
Vj = ui, for all 1 < j < rji. Using (4.17) and (4.21) with m = p, we obtain 

m 

(4.23) sup \fpit;ri)\<^p sup max/---/ sup Ia U ^*^''^'Kdui) , 

0<r<l 0<r<l -A |2|oo<l ^=1 

where 

(4.24) Ia= j gix + z)dAMx + z)... Oa^ hix + z) nidx) 



def 



and 6ip) < a < p. Fix a partition A = {Ai, . . . ,Aa} of type used in (4.17) 
and (4.23). Let /3 denote the number of 1-point sets in this partition. With- 
out loss of generality we can assume that Ai, . . . ,Ap are 1-point sets, and 
that . . . , Aa are 2-point sets. Then 

(4.25) dA^hix + z) = 2it{x-a + z,Qwj), for 1 < j < /?, 
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and 

(4.26) dAjh{x + z) = 2it{wj,Qwj), for (3 < j < a, 

with some Wj,Wj G {ui, . . . , }• Notice that |tfj|oo ^ 1 and luJjIoo < 1- 
Furthermore, using (4.25), (4.26) and (4.24), substituting 

(^x — a + z, Qwj^ = {x,Qwj^ — l^a — z,Qwj^, 

multiplying, applying the triangle inequality and re-enumerating Wj if neces- 
sary, we obtain 

(4.27) lA^p\tr ma^^Ip 
where 

(4.28) Ip= J gix + z) Y[{x,Qwj)B^idx)\, 

j=i I 

and 

(4.29) B= n {z-a,Qwj) f[ {wj,Qwj). 

j=p+i j=P+i 

Using \Qw\ < q\w\ <C(i q, we have 

(4.30) \B\ ^d,p q''~''{\zf-'' + \af-'') <p + H^-") 

since |2;|oo = T\ujn\oo < 1 (see (4.24)) and \wj\oo <^ 1, \wj\oo <^ 1. 
Let us split the measure /x = $ * jj,*^ {''if) as follows 

(4.31) 

^ = XO * XO = * ^(.;r)*('=-(f+i)[^/(f+l)!), X = H*^^'^^^'H-,rl 

where [u] denotes the integer part of u. Then we have 

(4.32) / U{x)ii{dx) = J U{xo-\ \- Xp+i) xo{dxo) x{dxi) ■ ■ ■ x{dxp+i) , 

for any integrable function U. Applying (4.32) to (4.28), writing x = xo-\ h 

Xp+i and using 

p p+i p+i 

il{x,Qwj)= J2 ■■■ H {xjiiQwi) ...{xj^.Qwp), 
j=i ji=0 jp=0 

we obtain 

(4.33) /p<p max Ipj, 

with 
(4.34) 

P+i 

Ipj = I I ■■■ I 9ix + z){xj^,Qwi) . . .{xjp,Qwp) B n Xidxs) Xoidxo)- 

s=l 
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Given the variables Xj^, . . . , xj^, p < p,we find among xi, . . . , Xp+i at least one 
variable, say xi, such that I ^ {ji, ■ ■ ■ ,jp}- Without loss of generality we can 
assume that 1 = 1. Then (4.34) yields 



(4.35) Ipj-^J---! {Qxj^,wi) ...{Qxj^,Wp) 
with 



p+i 

\B\ J xo{dxo) n X{dxs 

s=2 



J 



J g{x + z)x{dxi) 



X = XQ^ h Xp+i- 



Recall (see (4.4)) that g{x) = e[tQ[x — a] }. Therefore 



(4.36) 



J < sup 



/ e{tQ[x] + {L, x) } {dx; r) = V'(t) 



with V defined by (4.8). Hence, the bound (4.35) combined with (4.30) and 
(4.36) yields 

(4.37) 

with 



(4.38) 

We have 
(4.39) 



Jpj =!■■■! {Qxn,wi) ...{Qxj^, 



P+l 

Xo{dxo) n Xidxs)- 

s=2 



{Qxj^,wi) ...{Qxjp,Wp) 



since we assume that R> r, and since the variables Xg in the integral (4.38) 
satisfy |a;s|oo r + R<^ R. Combining (4.37)-(4.39), we get 

(4.40) Ipj <it i;{t)RPq''{l + |a|^-''). 
The estimate (4.40) combined with (4.33) and (4.27) yields 

(4.41) lA<-i^{t){\t\r'^Y q" r-'^'^ max i?'' (l + lal^"''). 

Using the condition R> r >\ and P + 2{a — (3) = 2a — P = p, p < P, we 

obtain 

(4.42) 

r-2° ma^ Rf (l + \af-P) ^r-^"" ma^ RP rf^-f (l + ( ^Y~''] 

0<P<I3 ^ ' ' / 0<p<f} \ \ r J J 



1 + ^) 



The inequalities P < p, q > 1 and 6{p) < a < p combined with (4.40)-(4.42) 
and (4.21)-(4.23) yield (4.10). 

The proof of (4.11) repeats the proof of (4.10) starting from (4.22) since 
now we have to estimate \Fj\ = \ fj\ (see (4.20)) instead of \fp\. In this proof 
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we have to use (4.20) instead of (4.21) and to replace everywhere p by j. 
Furthermore, instead of (4.31) we have to use a similar splitting of u of the 
form y = Xg* x*^'^^^ with 

(4.43) xo = **i^(-;r)*('^~^+^)['=/^+^)l) and x = ^*^'''^^^^^\-;r). 

In particular, the splitting (4.43) yields that the integral corresponding to J 
in (4.36) now satisfies 



J < sup 



/ e{tQ[x] + (L,x>}z/*['=/(P+i)](dx;r) = V'o(i), 
which leads to the factor in (4.11). □ 



5. The integration procedure for large \t\ 

Recall that 

(5.1) M{t-s) = {\t\s)~^l{\t\<s-^'^] + \t\l{\t\> s-^'^], 

where s > will be a positive large parameter. 

For a number T > 1 and a family of functions (^(•) = (^(•;s) : M M 
introduce 

(5.2) 7 = 7(s,r) =%up||(/?(i)| : s-^/2<^<2^|_ 

The following Theorem 5.1 sharpens Theorem 6.1 of [BG4] in cases where 
7 < 1. 

Theorem 5.1. Let f{t), t > denote a continuous function such that 
(p{0) = 1 and < ip < 1. Assume that, for some x > 4 and A > 1, 

(5.3) ip{t)ip{t + e) < AM'"{e;s), forallt>0 and e > 0. 
Let T >1. Assume that the number 7 defined by (5.2) satisfies 
(5.4) 

7 > 4"/("-^) s-"/^ if - 1< a < 0, 

^ (1 + In ^y^'^''^ > 4-/(-4) s-/' (1 + ln.)-/(-^), ifa = -1. 
Then the integral 

T 

J= J ip{t)t''dt 
5-1/2 

can be bounded as follows: 

(5.5) J<„,^ (^^y"^^"T"+i A, /or -l<a<0, 
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and 

J«x (^)'"^^''(l + ln A) (1+lnT) A, fora = -l. 

If (5.4) is not fulfilled then the following trivial bounds hold 
(5.6) 

J<„7r"+\ fora>-l, J<7(l + lns)(l + lnT), fora = -l. 

Proof. Evaluating the integral J using the method of Lcbcsgue integration 
by partitioning the range of ip in intervals [2~^~^, 2~'], we have to estimate the 
Lebcsguc measure of the corresponding sets 

Bi = {t: 2-'-i < ip{t) < 2-'} n K^/^T]. 

Using inequality (5.3), we shall show that two points in Bi are either very 
"close" or far apart. This means that the set Bi consists of 'small' clusters 
of size 01(8"^) separated by "large" gaps of size Oi{l). These constraints on 
the structure of Bi suffice to bound the measure of Bi well enough in order to 
estimate the size of J for k > A as claimed in Theorem 5.1. For trigonometric 
sums (p this condition translates to to the assumption that the dimension d 
satisfies d > 8. 

Throughout the proof we shall write <C instead of ^Q,,>f 
To prove (5.6) it suffices to use ^p{t) < 7 and to notice that 

/ — < (l + lns)(l + lnr), / t'^dt<^T'^+\ for a > -1. 

s-1/2 * g-1/2 

Let us prove (5.5). The inequality (5.3) implies that (set t = 0, use 
(p{0) = 1 and note that A > 1) 

(5.7) 99(t) < AA^"(t;s) and ip{t) ip{t + e) < M'^ie; s). 

Starting the proof of (5.5) with (5.7) we may assume without loss of gener- 
ality that A = 1, that is, that 

(5.8) ^(t) < M'' {t: s) and cp{t) ip{t + e) < M'^ie; s). 

Indeed, we may replace (p in (5.7) (resp. 7) by ip/A (resp. by 7/A), and we 
may integrate over (p/A instead of tp. Notice that now ip{0) < 1 and the case 
ip{0) < 1 is not excluded. 

Thus assuming (5.8) we have to prove that 

T 

(5.9) / ¥?(*) t° < 7^-4/" , for x > 4, 

with Fa = r°+i, for -1 < a < 0, and F_i = (1 + InT) (l + In A^ While 
proving (5.9) we may assume that 1 <^ s. Otherwise (5.9) obviously holds 
since < 7 < 1. 
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Let denote the smallest integer such that 2^'^ > 7. For the inte- 
gers I > Ij, introduce the sets 

Bi = [s-y^,T]n{t: 2-'-i<v^(t)<2-^}, 

Di = [s-y^T]n{t: ^(t)< 2-^-1}. 

Since the function (p satisfies < (p{t) < 7, for <t<T, the sets Bi and 

m 

Di are closed and Djn U [J Bi = [s"^/^, T]. Furthermore, (5.8) implies that 

(p{t) < t", for t > s'^^, whence Bi c [Li\T], where Li = 2('+i)/^. 

Recall that ip{t) < 2"', for t e Bi, and (p{t) < 2""^-^ for t e D^. There- 
fore the relation C T] yields 



1' r r 

(5.10) / ip{t)rdt< <f{t)edt + Y^ <p{t)rdt 

<2-"*G„ + ^ 2-' / eat, 

l = l-y 

where Ga = InT + Ins, for a = -1, and = r'*+\ for -1 < a < 0. We 
shall choose m such that 

(5.11) 2-"^ Ga < 7^-^/" F„ for - 1 < a < 0. 
More precisely, we choose the minimal m such that 

(5.12) m > In — , for - 1< a < 0. 

Using (5.10), (5.11), we see that the estimate (5.9) follows provided that we 
show that 

(5.13) £ < 1^'^/" , where Ii = 2"' / T dt. 

1=1-^ Bi 

Below we shall prove the inequalities 

(5.14) /;<(/ + InT) s'^ 2"'+^'/", for a = -1, 

II < T"+i s-^ 2-^+^^/", for - 1< a < 0, 

for I < m. These inequalities imply (5.13). Indeed, in both cases a = — 1 and 
a > — 1 we can apply the bound 

00 
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since k > A ensures the convergence of the series, and, according to the def- 
initions of 7 and Z-y, we have 2'~'t~^ < 7. In the case a = — 1 one needs in 
addition the following estimates. For Z-y ^ 1, we have 

1=1^ 1=0 
and, for Z-y > c^^ with a sufficiently large constant c^^, we obtain 

00 00 / 1 \ 

£ Z2-W-< J a;2-^+4^/^cZa;<Z^(2-'7)i-4/-<y-4/-M +lnlj. 

It remains to prove the inequalities (5.14). For the estimation of Ii we 
need a description of the structure of the sets Bi with I < m. Let t^t' € 5/ 
denote points such that t' > t. The inequality (5.8) and the definition of Bi 
imply 

(5.15) 4-'-^ < >l"(t' -t;s). 

If t' — t < s"""^/^ then by (5.15) and the definition of Ai{e; s) we get 

(5.16) t' -t<S, where 5 = 3-^4^+^^^''. 

li t' — t > then by (5.15) and the definition of Ai{e; s) we have 

(5.17) t'-t>p, where p = 4-(^+^)/". 
For X > 4 and sufficiently large s » 1 note that 

(5.18) S < P, provided Z < m. 

Indeed, using the definitions of 6 and p, we see that the inequality (5.18) follows 
from the inequality 4s~'^^^ < 2~"^, which is implied by (5.12), the assumption 
(5.4) and s > 1. 

The estimate (5.18) implies that either t' ~ t < 5 or t' — t > p. Therefore 
it follows from (5.16)-(5.18) that 

(5.19) teBi^Bin{t + S,t + p) = ^; 

that is, that in the interval {t+S,t+p) the function tp takes values lying outside 
of the interval [2"'-^ 2"']. 

Let us return to the proof of (5.14). If the set Bi is empty then (5.14) 
is obviously fulfilled. If Bi is nonempty then define ei = minjt : t G i?;}. 
Choosing t = e\ and using (5.19) we see that the interval (ei + 5, ei + p) does 
not intersect Bi. Similarly, let 62 denote the smallest t > ei + p such that 
t E Bi- Then the interval (e2 + (5, 62 + p) docs not intersect Bi. Repeating this 
procedure we construct a sequence L'^^ < ei < 62 < • • • < < T such that 

k 

(5.20) Bi C \J [ej,ej + S] and Cj+i > + p. 
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The sequence ei < • • • < cannot be infinite. Indeed, due to (5.20) we 

have 

r > efe > ei + (A; - l)p > L"^ + {k-l)p> kp, 

and therefore k <T/p. 

Using (5.20) we can finally prove (5.14). We start with the case a = — 1. 
Using ln(l + x) < x, for x > 0, we have 

k 

< 2^' ^ ^ < (/ + InT) 2-'+^'/'' s-^ 
since ei > Lf^ > p, k <T/p, and 

j: ±<Y: ^— < - E - « i^i^±i^ « (/ + inr)4^/-. 

Finally, let us prove (5.14) for — 1 < a < 0. We have 

k ^^^^ 

^ ^ E ^ ^ « ^ « .-2-^+^^/-T"+^ □ 

i=i j=i 
6. Trigonometric sums of irrational quadratic forms 

In this section we introduce a criterion for the rationality of a quadratic 

form in terms of the behavior of an associated sequence of trigonometric sums. 
Recall that a quadratic form Q[x\ = (^Qx,x'j, x € W^, with a nonzero symmet- 
ric matrix Q = (qij), ^ < i,j < d, is rational if there exists an M G M, M 7^ 0, 
such that the matrix MQ has integer entries; otherwise it is irrational. 
For a eW^, d> 1, consider the polynomial 

(6.1) P{x) = Q[x] + {a,x), xe M'^. 

Throughout this section we shall denote by p the uniform lattice measure 
in the box B{r) = {|x|oo < (cf. with the notation p{-;r) used in other 
sections). In other words, the measure p is nonncgative, normalized p{W^) = 

p(l/- f\ B{r)^ = 1 and assigns equal weights px *== A*({a^}) = (^M + l) to 
points X ^Ifi r\ B{r). 
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For k GN and i G M, introduce the trigonometric sum 

f{t) = mr)= J: e{tP{x)} n^'^'-'l =/^*(''=+^HW), 

where /j,*'' denotes the fe-fold convolution of /x. Let ju denote the symmetrization 
of fi, that is, J1{C) = ffi{C + x)fi{dx), for C € B'^. Writing = 
^ * * ji*^ and using the symmetrization inequahty (see Lemma 6.5 below), 
we obtain < | / (i) | < (p{t) with 

(6.2) ^{t) = ^{t-r)''^ E e{2i(gx,y)}/I,/i;^ 

For a family of functions g = g{-',r) : M — C parameterized by r > 0, 
consider the following condition: 

(6.3) for any < Sq < 6 < oo, lim sup \g(t;r) I = 0. 

''^'^ So<\t\<S 

We shall apply condition (6.3) to the trigonometric sums / and ip. See (6.4) 
and (6.6) for equivalent formulations of (6.3). The following characterization 
result holds without assumptions on the eigenvalues of Q. 

Theorem 6.1. Let k > 1. The quadratic form Q[x] is irrational if and 
only if (f satisfies condition (6.3). If Q is irrational then f satisfies (6.3). If a 
and Q are rational (that is, there exists AI ^ such that Ma and MQ have 
rational coordinates, resp. entries) then f does not satisfy (6.3). 

The proof of Theorem 6.1 will be given later. It is based on an application 
of the theory of successive minima (see Cassels [Ca], Davenport [Dav]) and 
techniques in [BGl]. 

For a given family of functions, the condition (6.3) allows the following 
equivalent reformulation: there exist functions (5o(r) J, and 5{r) ] oo such 
that 

(6.4) lim sup g{t; r) =0. 

'■^^ <5o(r)<|t|<<5(r) 

Applying a double large sieve type bound (see the estimate (5.22) in [BG4]) 
we obtain (p{t) <^d,k q^'^M-'^{t;r'^) with Ai defined by (1.21). Hence, assuming 
^G{r) > we have ip(t) <^d,k,Q ^ti'^)-, for r~^ <t< 5Q{r), and 

(6.5) lim sup sup (p{t) = 

aeftd r-i<t<<5o(r) 
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provided that the eigenvalues of Q are nonzero. Combining (6.4) and (6.5) we 
see that the irrationahty of Q is equivalent to the following condition: there 
exist (5(r) ] oo such that 

(6.6) lim sup sup ip{t) = 

provided that the eigenvalues of Q are nonzero. Due to |/p < ^p, relations 
(6.5) and (6.6) hold for / as well. 

Remark 6.2. An inspection of the proof shows that the condition (6.3) for 
the trigonometric sums / and (p holds uniformly over compact sets, say Q, of 
irrational matrices; that is, 

lim sup sup I ip{t] r) I =0. 
QeQ 5o<\t\<S 

Let us recall some facts of the theory of successive minima in the ge- 
ometry of numbers (see [Dav]). Let F : ^ [0, oo) be a norm, that is, 
F{ax) = \q\F{x), for a G R, and F{x + y) < F{x) +F{y). The successive 
minima Mi < • • • < of F with respect to the lattice are defined as fol- 
lows: Ml = inf{F(a;) : a; 7^ 0, x G Z*^}, and is defined as the lower bound 
of A > such that the set { a; G Z'' : F{x) < A } contains k linearly indepen- 
dent vectors. It is easy to see that there exist linearly independent vectors 
ai, . . . , G Z*^ such that F{aj) = Mj. 

Lemma 6.3 ([Dav, Lemma 3]). Let Lj{x) = Ylk=i ^jk^k, ^ ^ j ^ d, be 
linear forms in such that Xjk = Xkj- Assume that P > 1 and let \\9\\ 
denote the distance of the number 6 to the nearest integer. Then the number 
of x = {xi , Xd) G Z'' such that 

\\Lj{x)\\ < P~^, |x|oo < -P, foralll<j<d, 

is bounded from above by ^ ^'^ ^ , where Mi < • • • < Md are the first d of 

the 2d successive minima Mi < ■ ■ ■ < M2d of the norm F : M?'^ ^ [0, 00) 
defined for y = {x, m) G M^*^ with x, m G M'^ as 

(6.7) F{y) = ms^i^P\Li{x) - mi\, . . . ,P\Ld{x) - mdl P-^\x 

Furthermore., Mi > P"^. 

We shall use some well-known known bounds for trigonometric sums which 
go back to Weyl [We] . For our purposes we formulate a variation of such bounds 
as the following Lemma 6.4. Its proof is provided for the sake of completeness. 
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Lemma 6.4. For z = {zi, . . . , Zd) ^ M'^, 

9i^)= E e{<z,i/)}Mf E h{m), 

where 

d _, 

(6.8) h{m) = n ho{zj - 2Trmj), hois) = (l + r^s^) , s e Z. 

Proof. We have g{z) = v''{z) with 

2 d 

(6.9) ^^(2) = E e{ (2, y> } //y = E e{ (2;, y) } //^ = H 



2 

where n(x) = D^^j(x)/ (2r) and 2r = 2 [r] + 1 with the Dirichlct kernel 

[r] 

D[r]{x) = E exp{i,xn} = for x G M. 

n=— [r] 

Using I sinyl < min{l; \y\ } and <^ sin^(x/2), with y = xf and \x\ < tt, we 
obtain 

(6.10) tx(x) <min{^; 1} < ^-pi^, for < tt. 
The function u{x) is even and 27r-periodic. Hence (6.10) yields 

I< \x — 2'Km\ < 7r > 

(6.11) u\x) «fe E -r^ TT « E 



meZ ^l + r-2(a;-27rm)2^ m& ^1 + r-2 (x - 27rm)2^ 

where I{^} denotes the indicator function of the event A. Combining g^z) = 
v^{z) and (6.9), (6.11) we conclude the proof. □ 

Proof of Theorem 6.1. Let us show that for rational Q the trigonometric 
sum if does not satisfy (6.3). Assume for simplicity that Q has integer entries. 
Let jj, X v denote the product measure of measures /x and v. Clearly, {Qx,y^ 
G Z, for x, y G Z*^, and the function 

V{t)= Pme{2tm], pm'^= ]lx]l*^[[{x,y) el?'^ : {Qx,y) = m}), 

meZ 

with weights pm such that E Pm = 1, is a 7r-periodic function and ip{ir) = 1. 

Similar arguments show that for rational Q and a the trigonometric sum / 
does not satisfy (6.3). 

To complete the proof of the theorem, it remains to prove that if / or <^ 
does not satisfy condition (6.3) then Q is rational. The inequality |/p < 
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shows that we have to consider the case of only. Since (6.3) is not fulfilled, 
we have 

for some < do < 6 < oo, limsup sup (p(Trt;r) > 0. 

r-^oo So<\t\<S 

Hence, there exist £ > and sequences tn ^ to, Sq < \tn\ < S, and r„ j oo 
such that 

(6.12) fiTTtn, rn)>e> 0, for all G N. 

We shall prove that (6.12) implies the rationality of Q. 

Henceforth we shall write t and r instead of t„ and r„. Without loss 
of generality we shall assume throughout the proof that r = r„ is sufficiently 
large, that is, that r >c with some sufficiently large constant c = c{d, k, Q, S, e). 
Furthermore, we shall write <C instead of <^d,k,Q,5,e- 

The measure is concentrated in the cube B{2r) such that 
(6.13) 

G{x)iix= E G{x)iix, E /^x = 1, 0</ia;<r"'^, 

xel'^ xeB{2r)nZ'i xeB{2r)nZd 

for any summablc function G. Using the representation (6.2) of if, Lemma 
6.4, the relations (6.12) and (6.13) for the function h{m) defined by (6.8) with 
z = 27rtQx, we have 

(6.14) £<r-'^ E E h{m). 

Obviously we may replace h{m) in (6.14) by 
(6.15) 

h{m) Yl ho{zj — rrij), ho{s) = (l + r^s^) ^, z = z{x) = tQx, 
i=i 

since this yields in the inequality (6.14) an extra factor depending on d and k 
only. 

For vectors s = (si, . . . , s^) G Z'^ such that sj > and p = (pi, . . . , pd) 
G Z'^ such that pj = ±1, for all j, introduce the class Hg^p of vectors {x,m) 
G Z^^' such that 

X G B{2r), siga{zj{x) - mj) = pj, < \zj{x) - mj\ < ^^^^ , 

for all 1 < j < d. Here Zj{x) denote the j^^ coordinate of z{x) = tQx. 
Moreover, consider the class H of vectors {x, m) G Z^*^ such that 

(6.16) X G S(4r), \\zj{x)\\ < ^, for all 1 < j < d, 

where \\9\\ denotes the distance from the number 9 to the nearest integer. Then 

\jHs,p = {{x,m)eZ'^: xeB{2r)}. 

s,p 
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Furthermore, it is easy to see that (x , m) , {x ,m) G Hs,p imphes that 
{x — x,m — m) G H. Hence, card Hg p < card H. Using (6.14) and the 
definitions introduced, we obtain 

(6.17) e <€. ^ card Hs,p max{h{m) : {x,m) G -ffs,p} 

s,p 

d _f. 

< card H J2 11 (l + Sj) r~'^ card H, 

s j=l 

for sufficiently large r. 

Lemma 6.3 shows that the cardinality of the class H defined by (6.16) is 



norm F defined by (6.7) with P = Ar and 



bounded from above by — — , where M,- are the successive minima of the 

Ml • • • Md ' J 



d 

(6.18) Lj{x) = Zj{x) = {tQx)j = tQjiXi- 

i=l 

Combining (6.17) with card H < (Mi • • • Md)~^, we obtain r'^Mi • • • <C 1. 
Hence, using P = 4r and P"^ < Mi < • • • < M^, we have Mj x r~^, for all 
l<j<d. 

Let {as,bs) G Z^'^ denote linearly independent vectors such that 
F{{as,bs)) = Ms, for aU 1 < s < 2d. The equalities F{{as,bs)) = Mg to- 
gether with (6.7) and relations Mg x x P~^ imply that 

(6.19) I 

fls|oo*^l) |^s|oo*^l) l<S<(i. 

Combining the relations F(^{as,bs)) = Mg with P = 4r and (6.7), (6.18), 

we get (recall that t = tn and r = r^; therefore ag = ai"^ and bg = b^J^^ also 
depend on n) 



(6.20) Etn9,fe4?-6(;) 

fc=i 



l<j<d, l<s<d. 



The inequalities (6.19) guarantee that the sequence of systems { . . . , a^^^ } 
in Z*^ repeats infinitely often. Choosing an appropriate subsequence, we may 
assume that a^"'* = Ug are independent of n. Similarly we may assume that 
bi"^ = bg are independent of n. Passing to the limit as n — oo in (6.20) 
along the subsequence and using tn ^ to, to 7^ 0, we see that the numbers 
toqji, ■ ■ ■,toqjd satisfy 

d 

(6.21) J2 agitQqji = bgj, 1 < s < d, 

i=l 

for all 1 < j < d. Below we shall prove that the vectors ai , . . . , are 
linearly independent. Therefore the system (6.21) has the unique solution 
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toqji, . . . ,toqjd, which obviously has to be rational by Cramer's rule, for all 
l<j<d. 

To conclude the proof we have to show that the vectors ai, . . . , G Z'^ are 

linearly independent in Z'^ (or, equivalently, in W^). If ai, . . . ,0^ are linearly 

dependent then there exist f € M not all equal to zero such that \vs\ < 1 
d 

and f^sO-s = 0. Let us prove that there exist integers ui,...,Ud not all 

s=l 

def 

equal to zero such that \us\ <C 1 and x = ^ Ugag = By the multivariate 

s=l 

Dirichlet approximation (see, for example, [Ca, §V.10]), for any G N there 
exist Uj G Z and an integer < q < N such that 

(6.22) _ Jf± < _L_ , for all 1 < s < d. 



g 



The inequalities \vs\ < 1 and (6.22) yield \us\ < 2N. Since the vectors Ug have 

d 

integer coordinates and |as|oo ^ 1) the equation Vgag = together with 



d 

(6.22) implies x = Y, Ugag = 0, for sufficiently large AT ^ 1. Hence \ug\ <C 1. 

s=l 

For the vector (x,m) with m *== ^ Ugbg we have 

s=l 



(6.23) F{{x,m)) < E \ug\F{{ag,bg)) < - 

s=l 

since |tts| ^ 1 and F(^{ag,bs)) x 1/r. Using (6.7), we see that 

d 

(6.24) F{{x,m)) > 4r Yl Ughgj , for all 1 < j < d, 

s=l 

where bg = {bgi, . . . ,bsd)- Combining (6.23) and (6.24), and using that r is 

d d 

sufficiently large and that Y, '"s ^sj are integers, we conclude that Y '^s bgj = 0, 

s=l s=l 

d 

for all 1 < j < d. In other words, m = ^ u^fe^ = 0) which together with the 

d 

assumption x = ^ UsO^ = means that the vectors {ag^bg) G 7?'^, 1 < s < d, 

s=l 

are linearly dependent, a contradiction. □ 

A symmetrization inequality. The following symmetrization inequality is 
a generalization of a well-known classical inequality due to Weyl [We]. For a 
proof, see [BG4, Lemma 7.1]. Recall that the symmetrization ]! oi a measure 
H is defined by /1(C) = J n{C + x) fi{dx), for C G B'^. 
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Lemma 6.5. Let Q : —>■ be a linear symmetric operator, L G M"^ and 
C G M. Let Hi, fi2, fi3,f denote arbitrary probability measures on W^. Define a 
real valued polynomial of second order by 

P{x) = {Qx, x) + {L, x) + C, for x G R"^. 

Then the integral 

J = J e| t P(x) } /xi * /X2 * /X3 * i'{dx 
satisfies 2 J < Ji + J2, where 

-^1 = ne{'^t{Qx,y)}]li{dx)Ji2{dy), 

J2 = Jf e{2t {Qx,z)}]li{dx)Jl3{dz). 
In particular, if 112 = A*3 then J < J\. 



7. Properties of the distribution functions Fj 
and the signed measures Uj 

We start by establishing some properties of Fj and Uj (see (2.13)-(2.15) 
for definitions) using the Fourier transforms of the measures Vj. Recall that 

we denote 

W = [xeM.'^ : Q[x-a\el], I={a,(3\, Fj{I) = Fj{f3) - Fj{a). 
Let var Fj denote the variation of Fj. Write 

y = |x G M'^ : |M(a;/i?) - l| < w(ao)|, ao = {kr + l)/R, f=[r] + l/2. 

Lemma 7.1. The density D : ^ M. defined by (2.12) has continuous 
bounded partial derivatives d^D, for \a\oo < k — 2. 
Assume that j < k — 2. Then we have 

(7.1) \Dj{x) \ <^k,dr~^~'^ l{\x\oo < R + kf}, 
and 

(7.2) I Fj (s) I < var Fj <^k,d f {R/ff, for R>r. 
If the measure $ is defined by (2.17) then we have in addition 

(7.3) \Djix)\^k,dr-'-''l{xGV}, 
and 

(7.4) I Fj{I) I <fe,d r-^-'^ vol{W n V). 
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Proof. The Fourier transforms of complex valued functions / : M"* ^ C 
are denoted by 

7(y) = Ie{{y,x)} f{x)dx. 

Let u{x) = l{ |x|oo < 1/2}, X G W^, denote density of the measure tt. The 
measure has density Ur{x) =^ (2f) ~'^u(x/(2r)) . Write U = for 

density of the measure defined by (2.11). Then u has density D = U * 
(see (2.12)), where * denotes the convolution of functions. Therefore D = U v!^ 
and 

(7.5) \d^D{x) \ I \y''\\D{y)\dy < J \y''\\ur{y)\'' dy, 
since \U\ < 1. Hence, using 

we obtain 

(7.6) I d^D{x) I <fe,d r-l^li-ti, for |a|oo < k - 2, 

which proves the lemma's assertion about d'^D. 

Let us prove (7.1). Using the definition (2.13)-(2.14) of Dj, well-known 
properties of the Fourier transforms, and the equality j = ?7i + • • • + rim, 
estimating in (2.14) \us\ 1 and applying (7.5), (7.6), we obtain 

/» m 
• • • / I D(^\x)uf . . . I n 7r<''+'\dui) 

J ... J \D{y){y,u,y'...{y,Umy'^\dyf[7r<''+'\dui) 
<^k,d J \y\^\D{y)\dy^j,df- 



1=1 



Now (7.1) follows since D{x) = 0, for |x|oo > -R + kr. 

Let us prove (7.2). Using (2.15) and (7.1), we obtain 

I Fj{s) I < j\Dj{x) I dx <fc,d r-^-'^ (R + krf <^k,d r"^' {R/ff. 

Let us prove (7.3). By (2.21) and (2.22) the density D is constant outside 
the set V. Thus, outside V the derivatives of D vanish and Dj{x) = 0, for 
x(^W^\V. Thus (7.1) yields (7.3). 

Similarly, (7.4) follows from (7.1), (7.3) and the inequality 

\F{I)\< j'l\^xeWf\V^\Dj{x)\dx. □ 
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For further investigation of Fj we shall use the following double large sieve 
type bound (Lemma 7.2 below) which is a useful corollary of the large sieve of 
Bombieri and Iwaniec [BI]. It follows from Corollary 5.3 in [BG4] replacing in 
that corollary by q. 

Lemma 7.2. Assume that functions g,h : ^ C satisfy \g{x)\ < 1 

and \ h{x) | < 1. Let n and u denote arbitrary probability measures on such 
that 

IJ.(^{x eW^ : |x|oo<r}^=l and u(^{xeW^: \x\^< S}^ =1, 
for some T > and S > 0. Write 



J 



j(^J g{x)h{y)e{t{Qx,y)} n{dxj^ v{dy) 



t G 



Then there exists a positive constant ca, depending only on the dimension d, 
such that 



sup u \ X 

X' 



J'^aq\^^ST\t\f sup/.(x+ 

where S = |x G M*^ : |a;|oo < l}- 

Corollary 7.3. For an integer m > 2, the function 

^o(i) = sup / e{tQ[x] + (a,x) } v*"'{dx;f) 

satisfies 

(7.7) Mt) <<i min{ 1; (r^ \t\)-d/^ }. 



Proof. We shall derive the result using Lemma 7.2. 

If 1 > \t\ then (7.7) is obviously fulfilled since 'i/'o(i) < 1- Thus we can 
assume that |t| > 1. 

Using the obvious identity of the type 

J f{x) m * fi2 * Hsidx) = J f{x + y + z)iii{dx)ii2{dy)n'i{dz), 

the fact that the form Q[x\ is quadratic and Holder's inequality, we obtain 

(7.8) 



with 



J 



(t) < sup J J {dz; r) 



j(^J g{x)h{y)e{2t{Qx,y)} v{dx\r)^ v{dy\r] 



where the functions 1^1 < 1 and \h\ < 1 may depend on z and a. Choose 
S = T = f. The measure vi-'jf) has nonnegative density (with respect to the 
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Lebesgue measure on M'*) bounded from above by (2r) Therefore v{A] r) < 
(2r)~'^vol^, for any measurable set A C M.^. Thus Lemma 7.2 and the 
assumption \t\ > 1 yield 

whence, using (7.8), we derive (7.7). □ 

Lemma 7.4. The distribvMon function s Fq(s) has a bounded continu- 
ous derivative for d > 3. The functions s — > Fj(s) are functions of bounded 
variation for 2 < j < d/2, and 

sup|F,(.)|«,,,^(l+K)%.+^/2, 

Moreover, each of the functions s Fjis), j £ 2No, has [{d—l)/2]—j bounded 
continuous derivatives. 

Proof. Changing the variables t = Tr~^, we have 

(7.9) /(|i|r2)"min(l; (|i|r2)-''jdt«,,^ 4, for - 1< a < (3 - 1, 

(7.10) /(|t|r2)"min|l; (|t|r-2)-^| 1, for < a < /3. 

Using 6{j) < j, the bound (4.11) for | Fj(t)\, the estimate of Corollary 7.3 
for ipo{t) and (7.9), (7.10), choosing (3 = d/2 and a = 0{j), a = j respectively, 
we obtain 

(7.11) j\m\dt«,,, ^ (i+ K)%.+<i/2, 

for j G 2No such that j < — 1 + d/2. Similarly to (7.11), we obtain 

(7.12) i\m\^«j4^{^+^y<i'^'^ 

for j e 2N such that j < d/2. Finally we have 

(7.13) / I m\ f «j4 7^ (i + 

r\t\>l ' ' ^ 

for r > 1 and j G 2No such that j < d/2, and 

(7.14) / I*!'' I Fj{t)\ dt <oo, for < 5 < -j - 1 + d/2. 

R 

To conclude the proof it suffices to apply the Fourier and Fourier-Stieltjes 
inversion formulas and to use the bounds (7.11)-(7.14). □ 
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8. On the Lebesgue volumes of certain bodies 
related to indefinite quadratic forms 

We shall describe the asymptotic behavior of the volume of the set 

(8.1) A={xeR'^: M{x)eRIo, Q[x-a]el}, i? ^ oo, 

where M is the Minkowski functional of the set O defined by (1.13) and Iq and 
/ = [a, /3] are finite intervals. Throughout the section we assume that Q[x] is an 
indefinite quadratic form in M'', d> 3. Re-enumerating the eigenvalues of Q, 
choosing an appropriate orthonormal basis of and denoting the coordinates 
of X € M'' in this basis as xi, . . . ,Xd,, we may assume that 

Q[x] =qixl-\ \-qdxl 

with gi, . . . , g„ > 0, Qn+i, ■ ■ ■ ,Qd < ^ and some 1 < n < d/2. Indeed, if n > d/2, 
we may replace in (8.1) the matrix Q and the interval / by — Q and — / re- 
spectively. Let S = S'^~^ X denote a direct product of the unit (n— 1)- 
and {d — n — l)-dimensional spheres 

r = Xi + . . .Xj^ = 1 and p = + . . .x^ = 1 

with area elements drji and drj2 respectively, and drj =^ drji dr]2- Write 

Mo{x) = M{xi/^/\q^\,...,Xd/VWd\)- 

Lemma 8.1. The volume of the set A defined by (8.1) satisfies 

(8.2) lim i?-<^+2 vol A 

-R— >oo 

oo 

= I detQr^/2 j n'^-3( J l{Mo{urji + um) e } d??) du. 
s 

Proof. We shall use the following representation 

(8.3) volA = 2-ii2'^-2|detQ|-V2 J, 

where 

(8.4) J = B? J [ J ip{r, p)l{a< R^v < P} r"-i dv ) du. 

^-oo ^ 



(8.5) (p{r,p) = Jl{Mo{x + ao/R)elQ}dr], x = rr]i + pr]2, 
s 
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and ao = {\/\qi\ai, ^/\qd\ad), 
(8.6) 

r = r{u, v) = u, p = p{u, v) = \J — v, < u < oo, — oo <v < v? . 
For the proof of (8.3) it suffices to write 

voM = /l{M(a; + a) G Rh}\{a < Q{x\ </3}dx, 

to change the variables = Ryjj 1 < J < to use the polar coordinates 
r and p and to perform the change of variables (8.6). 

Let us prove (8.2). Assuming that R> |ao|, it is easy to see that 

I{ Mo(.T + ao/i?) G /o } = unless 0<r<c, 0<p<c, 

with some sufficiently large c = c(Mo,/o). Consequently, we have fir,p) = 
unless < u < c, 1^1 < c^. Therefore, for sufficiently large R, we can write 



c 



(8.7) J = Jf{u)du 



with 

f(u) =r''-^R'^ f ip{r,p)l{-oo <v <u'^} p'^-^'-'^dv. 

a/B? 

The function ip{r,p) is a continuous function of the variables u,v and qq/R. 
Therefore, for any u > 0, we obtain 

lim f{u) = u'^~^ {(3 -a) / I {Mo{ur]i + ur/2) € -To } drj. 

-R— >oo ^ 

The estimate f{u) c'^^^ (|a| + |/3|) allows us to apply the dominated con- 
vergence theorem, and (8.7) yields 



lim J = (/? - a) / u^~^ J I {Mo{urji + 'U772) G /q } c??? c^^t, 

which combined with (8.3) concludes the proof of (8.2). □ 
The relation (2.18) yields 

(8.8) {dq)~^^'^ \x\ < Mo{x) < m |a;|. 

Lemma 8.2. Let Iq = [0, A] and r = A + |ao|/-R, a = \/m - \ao\/R. Then 
the volume of the set A defined by (8.1) satisfies 

(8.9) vol A<^d{P- a) g('^-2)/2 
lfa>0 and \a\ + |/?| < R^/5 then 

(8.10) vol »d (/? - a) 
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Proof. We shall use the representation (8. 3)-(8. 6). Estimating 1 < |det(3| 
< g*^, we see that it suffices to prove that 

(8.11) J ^d{P-a)q^'^-^^/^T'^-\ 

(8.12) J>rf (/3-a)a'^-2. 

Let us prove (8.11). Using (8.8), wc have 

Mo{x + ao/R) > {dq)-^'^ {\x\ - \ao\/R) 

and 

(^(r, p)< / 1{ |a;p < dgr^ } drj <d l{ + < dq } 

s 

since r = u and |xp = + p"^. Hence, we get 

J «<i q('^-^y^ / ( / l{u^<dqT^} dv) du «d (/? - a) g(<^-2)/2 r'^-^, 
proving (8.11). 

Let us prove (8.12). Using (8.8), we have Mo(x+ao/i?) < m {\x\ + \ao\/ R) 

and 

f{r, p)> / 1{ |xp < 0-2 } dry >rf I{ 2 tt^ + I < } 
s 

since r = u and = — v < + Hence, using the condition |a| + |/3| < 
(T^ i?2/5^ obtain 

— OO 

• J l{a < vR^ < p} dv^ du (/?-«) 
proving (8.12). □ 
The definition (2.19) of the modulus of continuity a; of M yields 

(8.13) \Mo{x + y)-Mo{x)\ < \x\iu{d^\y\/\x\) . 

Write 
(8.14) 

£1,0=^, £2=-^ + -^, e4 = u}{2dei^o^/q), = U}{30de2 ^/q). 
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Lemma 8.3. Let lo = [1 - S,l + S], < 6 < 1/4. Assume that 

(8.15) £1,0 <c, 62 <c, e4<cq-^/\ £5 < cg-^/^ 

with some sufficiently small positive constant c = c(d, m) depending on d and 
m only. Then the volume of the set A defined by (8.1) satisfies 

(8.16) vol A (/3 - a) ((5 + £4 + £5) R"^-^ q(d-^y^. 

Proof. We shall write ^ instead of <^d,m- Using the representation (8.3)- 
(8.6) and the inequality | det Q |~^/^ < 1, we reduce the proof of (8.16) to the 
verification of 

(8.17) J<^iP-a){S + e^ + e5)q(^-^y^. 

Using = r^ + p^, applying the inequalities (8.8) and the triangle inequality, 
we see that 

ip{r, p) = unless 1 — S — £1,0 < |x| < -y/g (1 — 5) + £1,0 

since now Iq = [1 — S,l + S] with < S < 1/4. Hence the assumptions (8.15) 
and q> 1 imply that 

(8.18) v{r,p) = unless 1/2 < |x| < y^. 

The variable v in (8.4) satisfies I'ul < £2 < c < 1/8. Thus, (8.18) shows that we 
can assume that r = n in (8.4) satisfies u> 1/4. Using (8.18) we can estimate 
f-iP^ \x\ <S y^, and we have 

(8.19) J < g('^-3)/2i?2 jT ( J ^{r,p)dv)du. 

1/4 ^a/B? ' 

Using (8.13) and (8.18), we obtain 

(8.20) |mo(x+ ^) -Mo(x)| « < ^qe^- 

Let us write x = rtji + rr]2 + (p — r)ri2- The condition |x| > 1/2 yields 
r + p > 1/2. Therefore, using u = r > 1/4 and repeating the arguments used 
for the proof of (8.20), we obtain 

(8.21) ^ = -Jl-^ <8|i;| <8£2, \Mo{x)-uMo{vi + m)\<.VQ^5. 
Using (8.20) and (8.21), we can replace the indicator function in (8.5) by 

(8.22) I{l-S-co^/qe4-co^/qe5 < uMq (771+772) < 1+S+cq^ 84+00 ^/q 65}, 

where the constant cq = co{d,m) depends on d and m only. Now using 
(8.19), integrating the indicator function (8.22) in the variable u, estimating 
Mo{r]i + 772) 3> 1 and integrating in v, we obtain (8.17). □ 
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